2023-10-03T17:19:26.398 INFO:root:teuthology version: 0.0.1.dev194+g54e62bc 2023-10-03T17:19:26.398 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --description rbd:cli/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} features/layering msgr-failures/few objectstore/bluestore-comp-zlib pool/replicated-data-pool supported-random-distro$/{ubuntu_latest} workloads/rbd_support_module_recovery} --archive /home/teuthworker/archive/rraja-2023-10-03_17:10:36-rbd:cli-wip-ajarr-63009-only-reacquire-kickstart-distro-default-smithi/7409699 --owner scheduled_rraja@teuthology --name rraja-2023-10-03_17:10:36-rbd:cli-wip-ajarr-63009-only-reacquire-kickstart-distro-default-smithi -- /home/teuthworker/archive/rraja-2023-10-03_17:10:36-rbd:cli-wip-ajarr-63009-only-reacquire-kickstart-distro-default-smithi/7409699/orig.config.yaml 2023-10-03T17:19:26.427 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-03T17:19:26.502 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/rraja-2023-10-03_17:10:36-rbd:cli-wip-ajarr-63009-only-reacquire-kickstart-distro-default-smithi/7409699 branch: wip-ajarr-63009-only-reacquire-kickstart description: rbd:cli/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} features/layering msgr-failures/few objectstore/bluestore-comp-zlib pool/replicated-data-pool supported-random-distro$/{ubuntu_latest} workloads/rbd_support_module_recovery} email: rraja@redhat.com first_in_suite: false job_id: '7409699' kernel: kdb: true sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: rraja-2023-10-03_17:10:36-rbd:cli-wip-ajarr-63009-only-reacquire-kickstart-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 3 size: 30 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: wip-ajarr-63009-only-reacquire-kickstart ceph: conf: client: rbd default data pool: datapool rbd default features: 1 global: mon client directed command retry: 5 mon warn on pool no app: false ms inject socket failures: 5000 mgr: debug mgr: 20 debug ms: 1 debug rbd: 20 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: bluestore block size: 96636764160 bluestore compression algorithm: zlib bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME sha1: a7de91bfa5778a0d559a7e9bc8e35f93456b41ad ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log global: osd crush chooseleaf type: 0 osd pool default pg num: 128 osd pool default pgp num: 128 osd pool default size: 2 mon: osd default pool size: 2 install: ceph: flavor: default sha1: a7de91bfa5778a0d559a7e9bc8e35f93456b41ad thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-ajarr-63009-only-reacquire-kickstart sha1: a7de91bfa5778a0d559a7e9bc8e35f93456b41ad owner: scheduled_rraja@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 8213 sha1: a7de91bfa5778a0d559a7e9bc8e35f93456b41ad sleep_before_teardown: 0 suite: rbd:cli suite_branch: wip-ajarr-63009-only-reacquire-kickstart suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_a7de91bfa5778a0d559a7e9bc8e35f93456b41ad/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: a7de91bfa5778a0d559a7e9bc8e35f93456b41ad targets: smithi111.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEnvb/71JGXyR+c0QUpVhUkGSLg+nELPCl6VdIC0fmBzqofvlUwDqAZ884uaUDsJS88B0iMGAbcU5dgT0Qs4M/A= tasks: - install: null - ceph: null - exec: client.0: - sudo ceph osd pool create datapool 4 - rbd pool init datapool - install: extra_system_packages: - fio - workunit: clients: client.0: - rbd/rbd_support_module_recovery.sh teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 54e62bcbac4e53d9685e08328b790d3b20d71cae timestamp: 2023-10-03_17:10:36 tube: smithi user: rraja verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2830 2023-10-03T17:19:26.502 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_a7de91bfa5778a0d559a7e9bc8e35f93456b41ad/qa; will attempt to use it 2023-10-03T17:19:26.503 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_a7de91bfa5778a0d559a7e9bc8e35f93456b41ad/qa/tasks 2023-10-03T17:19:26.503 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-10-03T17:19:26.505 INFO:teuthology.task.internal:Checking packages... 2023-10-03T17:19:26.527 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash 'a7de91bfa5778a0d559a7e9bc8e35f93456b41ad' 2023-10-03T17:19:26.527 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-10-03T17:19:26.527 INFO:teuthology.packaging:ref: None 2023-10-03T17:19:26.527 INFO:teuthology.packaging:tag: None 2023-10-03T17:19:26.527 INFO:teuthology.packaging:branch: wip-ajarr-63009-only-reacquire-kickstart 2023-10-03T17:19:26.527 INFO:teuthology.packaging:sha1: a7de91bfa5778a0d559a7e9bc8e35f93456b41ad 2023-10-03T17:19:26.528 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=wip-ajarr-63009-only-reacquire-kickstart 2023-10-03T17:19:26.663 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-6336-ga7de91bf-1jammy 2023-10-03T17:19:26.663 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-10-03T17:19:26.670 INFO:teuthology.task.internal:no buildpackages task found 2023-10-03T17:19:26.670 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-10-03T17:19:26.682 INFO:teuthology.task.internal:Saving configuration 2023-10-03T17:19:26.693 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-10-03T17:19:26.700 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-03T17:19:26.723 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi111.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rraja-2023-10-03_17:10:36-rbd:cli-wip-ajarr-63009-only-reacquire-kickstart-distro-default-smithi/7409699', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-03 17:11:37.057710', 'locked_by': 'scheduled_rraja@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIEqE5frSkdXGJ3XtrYbNh1RDB10HpHr73ojqZdhWEyO9'} 2023-10-03T17:19:26.723 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-10-03T17:19:26.728 INFO:teuthology.task.internal:roles: ubuntu@smithi111.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2023-10-03T17:19:26.728 INFO:teuthology.run_tasks:Running task console_log... 2023-10-03T17:19:26.767 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f9143b63af0>, signals=[15]) 2023-10-03T17:19:26.767 INFO:teuthology.run_tasks:Running task internal.connect... 2023-10-03T17:19:26.774 INFO:teuthology.task.internal:Opening connections... 2023-10-03T17:19:26.774 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi111.front.sepia.ceph.com 2023-10-03T17:19:26.776 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi111.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-03T17:19:26.850 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-10-03T17:19:26.858 DEBUG:teuthology.orchestra.run.smithi111:> uname -m 2023-10-03T17:19:26.963 INFO:teuthology.orchestra.run.smithi111.stdout:x86_64 2023-10-03T17:19:26.963 DEBUG:teuthology.orchestra.run.smithi111:> cat /etc/os-release 2023-10-03T17:19:27.008 INFO:teuthology.orchestra.run.smithi111.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2023-10-03T17:19:27.008 INFO:teuthology.orchestra.run.smithi111.stdout:NAME="Ubuntu" 2023-10-03T17:19:27.008 INFO:teuthology.orchestra.run.smithi111.stdout:VERSION_ID="22.04" 2023-10-03T17:19:27.008 INFO:teuthology.orchestra.run.smithi111.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2023-10-03T17:19:27.008 INFO:teuthology.orchestra.run.smithi111.stdout:VERSION_CODENAME=jammy 2023-10-03T17:19:27.009 INFO:teuthology.orchestra.run.smithi111.stdout:ID=ubuntu 2023-10-03T17:19:27.009 INFO:teuthology.orchestra.run.smithi111.stdout:ID_LIKE=debian 2023-10-03T17:19:27.009 INFO:teuthology.orchestra.run.smithi111.stdout:HOME_URL="https://www.ubuntu.com/" 2023-10-03T17:19:27.009 INFO:teuthology.orchestra.run.smithi111.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2023-10-03T17:19:27.009 INFO:teuthology.orchestra.run.smithi111.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2023-10-03T17:19:27.009 INFO:teuthology.orchestra.run.smithi111.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2023-10-03T17:19:27.009 INFO:teuthology.orchestra.run.smithi111.stdout:UBUNTU_CODENAME=jammy 2023-10-03T17:19:27.010 INFO:teuthology.lock.ops:Updating smithi111.front.sepia.ceph.com on lock server 2023-10-03T17:19:27.032 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-10-03T17:19:27.042 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-10-03T17:19:27.047 INFO:teuthology.task.internal:Checking for old test directory... 2023-10-03T17:19:27.048 DEBUG:teuthology.orchestra.run.smithi111:> test '!' -e /home/ubuntu/cephtest 2023-10-03T17:19:27.056 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-10-03T17:19:27.062 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-10-03T17:19:27.062 DEBUG:teuthology.orchestra.run.smithi111:> test -z $(ls -A /var/lib/ceph) 2023-10-03T17:19:27.123 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-10-03T17:19:27.153 INFO:teuthology.run_tasks:Running task kernel... 2023-10-03T17:19:27.166 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-10-03T17:19:27.167 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-10-03T17:19:27.167 DEBUG:teuthology.orchestra.run.smithi111:> test -f /run/.containerenv -o -f /.dockerenv 2023-10-03T17:19:27.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-03T17:19:27.171 DEBUG:teuthology.orchestra.run.smithi111:> uname -r 2023-10-03T17:19:27.216 INFO:teuthology.orchestra.run.smithi111.stdout:5.15.0-56-generic 2023-10-03T17:19:27.216 INFO:teuthology.task.kernel:Running kernel on smithi111: 5.15.0-56-generic 2023-10-03T17:19:27.217 DEBUG:teuthology.orchestra.run.smithi111:> sudo apt-get clean 2023-10-03T17:19:27.347 DEBUG:teuthology.orchestra.run.smithi111:> sudo apt-get update 2023-10-03T17:19:27.483 INFO:teuthology.orchestra.run.smithi111.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2023-10-03T17:19:27.500 INFO:teuthology.orchestra.run.smithi111.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2023-10-03T17:19:27.560 INFO:teuthology.orchestra.run.smithi111.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB] 2023-10-03T17:19:27.601 INFO:teuthology.orchestra.run.smithi111.stdout:Get:4 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2023-10-03T17:19:27.743 INFO:teuthology.orchestra.run.smithi111.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [497 kB] 2023-10-03T17:19:27.805 INFO:teuthology.orchestra.run.smithi111.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,022 kB] 2023-10-03T17:19:27.831 INFO:teuthology.orchestra.run.smithi111.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [229 kB] 2023-10-03T17:19:27.836 INFO:teuthology.orchestra.run.smithi111.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [15.6 kB] 2023-10-03T17:19:27.836 INFO:teuthology.orchestra.run.smithi111.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [915 kB] 2023-10-03T17:19:27.846 INFO:teuthology.orchestra.run.smithi111.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [32.3 kB] 2023-10-03T17:19:27.847 INFO:teuthology.orchestra.run.smithi111.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [147 kB] 2023-10-03T17:19:27.849 INFO:teuthology.orchestra.run.smithi111.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [532 B] 2023-10-03T17:19:27.849 INFO:teuthology.orchestra.run.smithi111.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [657 kB] 2023-10-03T17:19:27.854 INFO:teuthology.orchestra.run.smithi111.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [987 kB] 2023-10-03T17:19:27.882 INFO:teuthology.orchestra.run.smithi111.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [216 kB] 2023-10-03T17:19:27.884 INFO:teuthology.orchestra.run.smithi111.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [21.9 kB] 2023-10-03T17:19:27.885 INFO:teuthology.orchestra.run.smithi111.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [3,888 B] 2023-10-03T17:19:27.885 INFO:teuthology.orchestra.run.smithi111.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [41.6 kB] 2023-10-03T17:19:27.886 INFO:teuthology.orchestra.run.smithi111.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [9,768 B] 2023-10-03T17:19:27.886 INFO:teuthology.orchestra.run.smithi111.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B] 2023-10-03T17:19:27.886 INFO:teuthology.orchestra.run.smithi111.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [41.7 kB] 2023-10-03T17:19:27.887 INFO:teuthology.orchestra.run.smithi111.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [33.9 kB] 2023-10-03T17:19:27.887 INFO:teuthology.orchestra.run.smithi111.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [10.5 kB] 2023-10-03T17:19:27.901 INFO:teuthology.orchestra.run.smithi111.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2023-10-03T17:19:27.917 INFO:teuthology.orchestra.run.smithi111.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [24.3 kB] 2023-10-03T17:19:27.918 INFO:teuthology.orchestra.run.smithi111.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [13.4 kB] 2023-10-03T17:19:27.918 INFO:teuthology.orchestra.run.smithi111.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.4 kB] 2023-10-03T17:19:27.918 INFO:teuthology.orchestra.run.smithi111.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [640 B] 2023-10-03T17:19:28.031 INFO:teuthology.orchestra.run.smithi111.stdout:Get:29 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [331 kB] 2023-10-03T17:19:28.288 INFO:teuthology.orchestra.run.smithi111.stdout:Get:30 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [812 kB] 2023-10-03T17:19:28.429 INFO:teuthology.orchestra.run.smithi111.stdout:Get:31 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [170 kB] 2023-10-03T17:19:28.450 INFO:teuthology.orchestra.run.smithi111.stdout:Get:32 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.3 kB] 2023-10-03T17:19:28.452 INFO:teuthology.orchestra.run.smithi111.stdout:Get:33 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [32.0 kB] 2023-10-03T17:19:28.453 INFO:teuthology.orchestra.run.smithi111.stdout:Get:34 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [899 kB] 2023-10-03T17:19:28.516 INFO:teuthology.orchestra.run.smithi111.stdout:Get:35 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [145 kB] 2023-10-03T17:19:28.525 INFO:teuthology.orchestra.run.smithi111.stdout:Get:36 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [532 B] 2023-10-03T17:19:28.525 INFO:teuthology.orchestra.run.smithi111.stdout:Get:37 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [785 kB] 2023-10-03T17:19:28.561 INFO:teuthology.orchestra.run.smithi111.stdout:Get:38 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [559 kB] 2023-10-03T17:19:28.744 INFO:teuthology.orchestra.run.smithi111.stdout:Get:39 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [144 kB] 2023-10-03T17:19:29.124 INFO:teuthology.orchestra.run.smithi111.stdout:Get:40 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.7 kB] 2023-10-03T17:19:29.131 INFO:teuthology.orchestra.run.smithi111.stdout:Get:41 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [36.5 kB] 2023-10-03T17:19:29.147 INFO:teuthology.orchestra.run.smithi111.stdout:Get:42 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,032 B] 2023-10-03T17:19:29.147 INFO:teuthology.orchestra.run.smithi111.stdout:Get:43 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,060 B] 2023-10-03T17:19:29.150 INFO:teuthology.orchestra.run.smithi111.stdout:Get:44 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2023-10-03T17:19:33.419 INFO:teuthology.orchestra.run.smithi111.stdout:Fetched 9,225 kB in 2s (5,305 kB/s) 2023-10-03T17:19:34.676 INFO:teuthology.orchestra.run.smithi111.stdout:Reading package lists... 2023-10-03T17:19:34.693 DEBUG:teuthology.orchestra.run.smithi111:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-10-03T17:19:34.753 INFO:teuthology.orchestra.run.smithi111.stdout:Reading package lists... 2023-10-03T17:19:34.937 INFO:teuthology.orchestra.run.smithi111.stdout:Building dependency tree... 2023-10-03T17:19:34.938 INFO:teuthology.orchestra.run.smithi111.stdout:Reading state information... 2023-10-03T17:19:35.093 INFO:teuthology.orchestra.run.smithi111.stdout:The following packages were automatically installed and are no longer required: 2023-10-03T17:19:35.093 INFO:teuthology.orchestra.run.smithi111.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-10-03T17:19:35.093 INFO:teuthology.orchestra.run.smithi111.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-10-03T17:19:35.094 INFO:teuthology.orchestra.run.smithi111.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-10-03T17:19:35.094 INFO:teuthology.orchestra.run.smithi111.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2023-10-03T17:19:35.094 INFO:teuthology.orchestra.run.smithi111.stdout: python2.7-minimal 2023-10-03T17:19:35.094 INFO:teuthology.orchestra.run.smithi111.stdout:Use 'sudo apt autoremove' to remove them. 2023-10-03T17:19:35.095 INFO:teuthology.orchestra.run.smithi111.stdout:The following additional packages will be installed: 2023-10-03T17:19:35.095 INFO:teuthology.orchestra.run.smithi111.stdout: linux-generic linux-headers-5.15.0-84 linux-headers-5.15.0-84-generic 2023-10-03T17:19:35.095 INFO:teuthology.orchestra.run.smithi111.stdout: linux-headers-generic linux-image-5.15.0-84-generic 2023-10-03T17:19:35.095 INFO:teuthology.orchestra.run.smithi111.stdout: linux-modules-5.15.0-84-generic linux-modules-extra-5.15.0-84-generic 2023-10-03T17:19:35.097 INFO:teuthology.orchestra.run.smithi111.stdout:Suggested packages: 2023-10-03T17:19:35.097 INFO:teuthology.orchestra.run.smithi111.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2023-10-03T17:19:35.097 INFO:teuthology.orchestra.run.smithi111.stdout:Recommended packages: 2023-10-03T17:19:35.097 INFO:teuthology.orchestra.run.smithi111.stdout: thermald 2023-10-03T17:19:35.133 INFO:teuthology.orchestra.run.smithi111.stdout:The following NEW packages will be installed: 2023-10-03T17:19:35.133 INFO:teuthology.orchestra.run.smithi111.stdout: linux-headers-5.15.0-84 linux-headers-5.15.0-84-generic 2023-10-03T17:19:35.133 INFO:teuthology.orchestra.run.smithi111.stdout: linux-image-5.15.0-84-generic linux-modules-5.15.0-84-generic 2023-10-03T17:19:35.133 INFO:teuthology.orchestra.run.smithi111.stdout: linux-modules-extra-5.15.0-84-generic 2023-10-03T17:19:35.134 INFO:teuthology.orchestra.run.smithi111.stdout:The following packages will be upgraded: 2023-10-03T17:19:35.135 INFO:teuthology.orchestra.run.smithi111.stdout: linux-generic linux-headers-generic linux-image-generic 2023-10-03T17:19:35.184 INFO:teuthology.orchestra.run.smithi111.stdout:3 upgraded, 5 newly installed, 0 to remove and 266 not upgraded. 2023-10-03T17:19:35.184 INFO:teuthology.orchestra.run.smithi111.stdout:Need to get 113 MB of archives. 2023-10-03T17:19:35.185 INFO:teuthology.orchestra.run.smithi111.stdout:After this operation, 582 MB of additional disk space will be used. 2023-10-03T17:19:35.185 INFO:teuthology.orchestra.run.smithi111.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-84-generic amd64 5.15.0-84.93 [22.5 MB] 2023-10-03T17:19:35.610 INFO:teuthology.orchestra.run.smithi111.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-84-generic amd64 5.15.0-84.93 [11.5 MB] 2023-10-03T17:19:35.708 INFO:teuthology.orchestra.run.smithi111.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-84-generic amd64 5.15.0-84.93 [63.9 MB] 2023-10-03T17:19:36.484 INFO:teuthology.orchestra.run.smithi111.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.84.81 [1,698 B] 2023-10-03T17:19:36.484 INFO:teuthology.orchestra.run.smithi111.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.84.81 [2,476 B] 2023-10-03T17:19:36.485 INFO:teuthology.orchestra.run.smithi111.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-84 all 5.15.0-84.93 [12.3 MB] 2023-10-03T17:19:36.601 INFO:teuthology.orchestra.run.smithi111.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-84-generic amd64 5.15.0-84.93 [2,853 kB] 2023-10-03T17:19:36.621 INFO:teuthology.orchestra.run.smithi111.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.84.81 [2,330 B] 2023-10-03T17:19:37.021 INFO:teuthology.orchestra.run.smithi111.stdout:Fetched 113 MB in 1s (76.4 MB/s) 2023-10-03T17:19:37.188 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package linux-modules-5.15.0-84-generic. 2023-10-03T17:19:39.172 INFO:teuthology.orchestra.run.smithi111.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2023-10-03T17:19:39.177 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../0-linux-modules-5.15.0-84-generic_5.15.0-84.93_amd64.deb ... 2023-10-03T17:19:39.243 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking linux-modules-5.15.0-84-generic (5.15.0-84.93) ... 2023-10-03T17:19:41.362 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package linux-image-5.15.0-84-generic. 2023-10-03T17:19:41.384 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../1-linux-image-5.15.0-84-generic_5.15.0-84.93_amd64.deb ... 2023-10-03T17:19:41.481 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking linux-image-5.15.0-84-generic (5.15.0-84.93) ... 2023-10-03T17:19:41.965 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-84-generic. 2023-10-03T17:19:41.986 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-84-generic_5.15.0-84.93_amd64.deb ... 2023-10-03T17:19:42.023 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking linux-modules-extra-5.15.0-84-generic (5.15.0-84.93) ... 2023-10-03T17:19:48.675 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../3-linux-generic_5.15.0.84.81_amd64.deb ... 2023-10-03T17:19:48.785 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking linux-generic (5.15.0.84.81) over (5.15.0.56.54) ... 2023-10-03T17:19:49.118 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.84.81_amd64.deb ... 2023-10-03T17:19:49.239 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking linux-image-generic (5.15.0.84.81) over (5.15.0.56.54) ... 2023-10-03T17:19:49.526 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package linux-headers-5.15.0-84. 2023-10-03T17:19:49.548 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../5-linux-headers-5.15.0-84_5.15.0-84.93_all.deb ... 2023-10-03T17:19:49.583 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking linux-headers-5.15.0-84 (5.15.0-84.93) ... 2023-10-03T17:19:55.641 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package linux-headers-5.15.0-84-generic. 2023-10-03T17:19:55.679 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../6-linux-headers-5.15.0-84-generic_5.15.0-84.93_amd64.deb ... 2023-10-03T17:19:55.716 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking linux-headers-5.15.0-84-generic (5.15.0-84.93) ... 2023-10-03T17:19:58.473 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.84.81_amd64.deb ... 2023-10-03T17:19:58.598 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking linux-headers-generic (5.15.0.84.81) over (5.15.0.56.54) ... 2023-10-03T17:19:58.936 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up linux-headers-5.15.0-84 (5.15.0-84.93) ... 2023-10-03T17:19:59.058 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up linux-headers-5.15.0-84-generic (5.15.0-84.93) ... 2023-10-03T17:19:59.184 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up linux-headers-generic (5.15.0.84.81) ... 2023-10-03T17:19:59.313 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up linux-image-5.15.0-84-generic (5.15.0-84.93) ... 2023-10-03T17:20:00.647 INFO:teuthology.orchestra.run.smithi111.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2023-10-03T17:20:00.647 INFO:teuthology.orchestra.run.smithi111.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2023-10-03T17:20:00.647 INFO:teuthology.orchestra.run.smithi111.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-84-generic 2023-10-03T17:20:00.648 INFO:teuthology.orchestra.run.smithi111.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-84-generic 2023-10-03T17:20:00.807 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up linux-modules-extra-5.15.0-84-generic (5.15.0-84.93) ... 2023-10-03T17:20:02.233 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up linux-modules-5.15.0-84-generic (5.15.0-84.93) ... 2023-10-03T17:20:03.843 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up linux-image-generic (5.15.0.84.81) ... 2023-10-03T17:20:03.969 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up linux-generic (5.15.0.84.81) ... 2023-10-03T17:20:04.095 INFO:teuthology.orchestra.run.smithi111.stdout:Processing triggers for linux-image-5.15.0-84-generic (5.15.0-84.93) ... 2023-10-03T17:20:04.160 INFO:teuthology.orchestra.run.smithi111.stdout:/etc/kernel/postinst.d/initramfs-tools: 2023-10-03T17:20:04.160 INFO:teuthology.orchestra.run.smithi111.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-84-generic 2023-10-03T17:20:18.984 INFO:teuthology.orchestra.run.smithi111.stdout:/etc/kernel/postinst.d/zz-update-grub: 2023-10-03T17:20:18.984 INFO:teuthology.orchestra.run.smithi111.stdout:Sourcing file `/etc/default/grub' 2023-10-03T17:20:19.011 INFO:teuthology.orchestra.run.smithi111.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-10-03T17:20:19.026 INFO:teuthology.orchestra.run.smithi111.stdout:Generating grub configuration file ... 2023-10-03T17:20:19.338 INFO:teuthology.orchestra.run.smithi111.stdout:Found linux image: /boot/vmlinuz-5.15.0-84-generic 2023-10-03T17:20:19.359 INFO:teuthology.orchestra.run.smithi111.stdout:Found initrd image: /boot/initrd.img-5.15.0-84-generic 2023-10-03T17:20:19.667 INFO:teuthology.orchestra.run.smithi111.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2023-10-03T17:20:19.673 INFO:teuthology.orchestra.run.smithi111.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2023-10-03T17:20:19.744 INFO:teuthology.orchestra.run.smithi111.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2023-10-03T17:20:19.749 INFO:teuthology.orchestra.run.smithi111.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2023-10-03T17:20:19.920 INFO:teuthology.orchestra.run.smithi111.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2023-10-03T17:20:19.920 INFO:teuthology.orchestra.run.smithi111.stdout:Systems on them will not be added to the GRUB boot configuration. 2023-10-03T17:20:19.920 INFO:teuthology.orchestra.run.smithi111.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2023-10-03T17:20:19.972 INFO:teuthology.orchestra.run.smithi111.stdout:done 2023-10-03T17:20:20.435 DEBUG:teuthology.orchestra.run.smithi111:> dpkg -s linux-image-generic 2023-10-03T17:20:20.495 INFO:teuthology.orchestra.run.smithi111.stdout:Package: linux-image-generic 2023-10-03T17:20:20.495 INFO:teuthology.orchestra.run.smithi111.stdout:Status: install ok installed 2023-10-03T17:20:20.495 INFO:teuthology.orchestra.run.smithi111.stdout:Priority: optional 2023-10-03T17:20:20.495 INFO:teuthology.orchestra.run.smithi111.stdout:Section: kernel 2023-10-03T17:20:20.495 INFO:teuthology.orchestra.run.smithi111.stdout:Installed-Size: 20 2023-10-03T17:20:20.495 INFO:teuthology.orchestra.run.smithi111.stdout:Maintainer: Ubuntu Kernel Team 2023-10-03T17:20:20.496 INFO:teuthology.orchestra.run.smithi111.stdout:Architecture: amd64 2023-10-03T17:20:20.496 INFO:teuthology.orchestra.run.smithi111.stdout:Source: linux-meta 2023-10-03T17:20:20.496 INFO:teuthology.orchestra.run.smithi111.stdout:Version: 5.15.0.84.81 2023-10-03T17:20:20.496 INFO:teuthology.orchestra.run.smithi111.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.1), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-84), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.1) 2023-10-03T17:20:20.496 INFO:teuthology.orchestra.run.smithi111.stdout:Depends: linux-image-5.15.0-84-generic, linux-modules-extra-5.15.0-84-generic, linux-firmware, intel-microcode, amd64-microcode 2023-10-03T17:20:20.496 INFO:teuthology.orchestra.run.smithi111.stdout:Recommends: thermald 2023-10-03T17:20:20.496 INFO:teuthology.orchestra.run.smithi111.stdout:Description: Generic Linux kernel image 2023-10-03T17:20:20.496 INFO:teuthology.orchestra.run.smithi111.stdout: This package will always depend on the latest generic kernel image 2023-10-03T17:20:20.496 INFO:teuthology.orchestra.run.smithi111.stdout: available. 2023-10-03T17:20:20.497 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-84-generic 2023-10-03T17:20:20.497 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-10-03T17:20:20.497 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-10-03T17:20:20.497 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2023-10-03T17:20:20.497 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi111.front.sepia.ceph.com, path=None, version=distro) 2023-10-03T17:20:20.497 DEBUG:teuthology.orchestra.run.smithi111:> sudo apt-get clean 2023-10-03T17:20:20.640 DEBUG:teuthology.orchestra.run.smithi111:> sudo apt-get update 2023-10-03T17:20:20.770 INFO:teuthology.orchestra.run.smithi111.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2023-10-03T17:20:20.780 INFO:teuthology.orchestra.run.smithi111.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2023-10-03T17:20:20.799 INFO:teuthology.orchestra.run.smithi111.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2023-10-03T17:20:26.881 INFO:teuthology.orchestra.run.smithi111.stdout:Hit:4 http://security.ubuntu.com/ubuntu jammy-security InRelease 2023-10-03T17:20:27.917 INFO:teuthology.orchestra.run.smithi111.stdout:Reading package lists... 2023-10-03T17:20:27.935 DEBUG:teuthology.orchestra.run.smithi111:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-10-03T17:20:27.993 INFO:teuthology.orchestra.run.smithi111.stdout:Reading package lists... 2023-10-03T17:20:28.179 INFO:teuthology.orchestra.run.smithi111.stdout:Building dependency tree... 2023-10-03T17:20:28.181 INFO:teuthology.orchestra.run.smithi111.stdout:Reading state information... 2023-10-03T17:20:28.345 INFO:teuthology.orchestra.run.smithi111.stdout:linux-image-generic is already the newest version (5.15.0.84.81). 2023-10-03T17:20:28.345 INFO:teuthology.orchestra.run.smithi111.stdout:The following packages were automatically installed and are no longer required: 2023-10-03T17:20:28.345 INFO:teuthology.orchestra.run.smithi111.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-10-03T17:20:28.346 INFO:teuthology.orchestra.run.smithi111.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-10-03T17:20:28.346 INFO:teuthology.orchestra.run.smithi111.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-10-03T17:20:28.346 INFO:teuthology.orchestra.run.smithi111.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2023-10-03T17:20:28.346 INFO:teuthology.orchestra.run.smithi111.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2023-10-03T17:20:28.346 INFO:teuthology.orchestra.run.smithi111.stdout:Use 'sudo apt autoremove' to remove them. 2023-10-03T17:20:28.383 INFO:teuthology.orchestra.run.smithi111.stdout:0 upgraded, 0 newly installed, 0 to remove and 266 not upgraded. 2023-10-03T17:20:28.385 DEBUG:teuthology.orchestra.run.smithi111:> dpkg -s linux-image-generic 2023-10-03T17:20:28.401 INFO:teuthology.orchestra.run.smithi111.stdout:Package: linux-image-generic 2023-10-03T17:20:28.401 INFO:teuthology.orchestra.run.smithi111.stdout:Status: install ok installed 2023-10-03T17:20:28.401 INFO:teuthology.orchestra.run.smithi111.stdout:Priority: optional 2023-10-03T17:20:28.401 INFO:teuthology.orchestra.run.smithi111.stdout:Section: kernel 2023-10-03T17:20:28.401 INFO:teuthology.orchestra.run.smithi111.stdout:Installed-Size: 20 2023-10-03T17:20:28.401 INFO:teuthology.orchestra.run.smithi111.stdout:Maintainer: Ubuntu Kernel Team 2023-10-03T17:20:28.401 INFO:teuthology.orchestra.run.smithi111.stdout:Architecture: amd64 2023-10-03T17:20:28.401 INFO:teuthology.orchestra.run.smithi111.stdout:Source: linux-meta 2023-10-03T17:20:28.401 INFO:teuthology.orchestra.run.smithi111.stdout:Version: 5.15.0.84.81 2023-10-03T17:20:28.402 INFO:teuthology.orchestra.run.smithi111.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.1), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-84), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.1) 2023-10-03T17:20:28.402 INFO:teuthology.orchestra.run.smithi111.stdout:Depends: linux-image-5.15.0-84-generic, linux-modules-extra-5.15.0-84-generic, linux-firmware, intel-microcode, amd64-microcode 2023-10-03T17:20:28.402 INFO:teuthology.orchestra.run.smithi111.stdout:Recommends: thermald 2023-10-03T17:20:28.402 INFO:teuthology.orchestra.run.smithi111.stdout:Description: Generic Linux kernel image 2023-10-03T17:20:28.402 INFO:teuthology.orchestra.run.smithi111.stdout: This package will always depend on the latest generic kernel image 2023-10-03T17:20:28.402 INFO:teuthology.orchestra.run.smithi111.stdout: available. 2023-10-03T17:20:28.402 DEBUG:teuthology.orchestra.run.smithi111:> mktemp 2023-10-03T17:20:28.448 INFO:teuthology.orchestra.run.smithi111.stdout:/tmp/tmp.57gRoQpcS3 2023-10-03T17:20:28.448 DEBUG:teuthology.orchestra.run.smithi111:> sudo cp /boot/grub/grub.cfg /tmp/tmp.57gRoQpcS3 2023-10-03T17:20:28.504 DEBUG:teuthology.orchestra.run.smithi111:> sudo chmod 0666 /tmp/tmp.57gRoQpcS3 2023-10-03T17:20:28.624 DEBUG:teuthology.orchestra.remote:smithi111:/tmp/tmp.57gRoQpcS3 is 10KB 2023-10-03T17:20:28.638 DEBUG:teuthology.orchestra.run.smithi111:> rm -fr /tmp/tmp.57gRoQpcS3 2023-10-03T17:20:28.643 DEBUG:teuthology.orchestra.run.smithi111:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2023-10-03T17:20:28.695 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-10-03T17:20:28.695 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd of=/etc/grub.d/01_ceph_kernel 2023-10-03T17:20:28.696 DEBUG:teuthology.orchestra.run.smithi111:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2023-10-03T17:20:28.763 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-84-generic 2023-10-03T17:20:28.763 DEBUG:teuthology.orchestra.run.smithi111:> sudo update-grub 2023-10-03T17:20:29.868 INFO:teuthology.orchestra.run.smithi111.stderr:Sourcing file `/etc/default/grub' 2023-10-03T17:20:29.869 INFO:teuthology.orchestra.run.smithi111.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-10-03T17:20:29.872 INFO:teuthology.orchestra.run.smithi111.stderr:Generating grub configuration file ... 2023-10-03T17:20:30.101 INFO:teuthology.orchestra.run.smithi111.stderr:Found linux image: /boot/vmlinuz-5.15.0-84-generic 2023-10-03T17:20:30.108 INFO:teuthology.orchestra.run.smithi111.stderr:Found initrd image: /boot/initrd.img-5.15.0-84-generic 2023-10-03T17:20:30.342 INFO:teuthology.orchestra.run.smithi111.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2023-10-03T17:20:30.346 INFO:teuthology.orchestra.run.smithi111.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2023-10-03T17:20:30.386 INFO:teuthology.orchestra.run.smithi111.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2023-10-03T17:20:30.391 INFO:teuthology.orchestra.run.smithi111.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2023-10-03T17:20:30.505 INFO:teuthology.orchestra.run.smithi111.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2023-10-03T17:20:30.505 INFO:teuthology.orchestra.run.smithi111.stderr:Systems on them will not be added to the GRUB boot configuration. 2023-10-03T17:20:30.505 INFO:teuthology.orchestra.run.smithi111.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2023-10-03T17:20:30.523 INFO:teuthology.orchestra.run.smithi111.stderr:done 2023-10-03T17:20:30.525 DEBUG:teuthology.orchestra.run.smithi111:> sudo shutdown -r now 2023-10-03T17:21:00.546 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2023-10-03T17:21:00.547 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi111.front.sepia.ceph.com' 2023-10-03T17:21:00.548 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi111.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-03T17:21:18.970 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.111 2023-10-03T17:21:27.980 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi111.front.sepia.ceph.com' 2023-10-03T17:21:27.981 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi111.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-03T17:21:31.038 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.111 2023-10-03T17:21:43.050 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi111.front.sepia.ceph.com' 2023-10-03T17:21:43.051 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi111.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-03T17:21:46.106 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.111 2023-10-03T17:22:01.119 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi111.front.sepia.ceph.com' 2023-10-03T17:22:01.120 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi111.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-03T17:22:19.646 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.111 2023-10-03T17:22:37.665 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi111.front.sepia.ceph.com' 2023-10-03T17:22:37.666 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi111.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-03T17:22:37.899 DEBUG:teuthology.orchestra.run.smithi111:> true 2023-10-03T17:22:38.531 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi111.front.sepia.ceph.com' 2023-10-03T17:22:38.532 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.15.0-84-generic"... 2023-10-03T17:22:38.532 DEBUG:teuthology.orchestra.run.smithi111:> uname -r 2023-10-03T17:22:38.580 INFO:teuthology.orchestra.run.smithi111.stdout:5.15.0-84-generic 2023-10-03T17:22:38.580 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-84-generic vs 5.15.0-84-generic 2023-10-03T17:22:38.580 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-10-03T17:22:38.580 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2023-10-03T17:22:39.581 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2023-10-03T17:22:39.582 DEBUG:teuthology.orchestra.run.smithi111:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-10-03T17:22:39.659 INFO:teuthology.orchestra.run.smithi111.stdout:ttyS1 2023-10-03T17:22:39.701 DEBUG:teuthology.parallel:result is None 2023-10-03T17:22:39.701 INFO:teuthology.run_tasks:Running task internal.base... 2023-10-03T17:22:39.709 INFO:teuthology.task.internal:Creating test directory... 2023-10-03T17:22:39.710 DEBUG:teuthology.orchestra.run.smithi111:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-10-03T17:22:39.716 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-10-03T17:22:39.723 INFO:teuthology.run_tasks:Running task internal.archive... 2023-10-03T17:22:39.729 INFO:teuthology.task.internal:Creating archive directory... 2023-10-03T17:22:39.729 DEBUG:teuthology.orchestra.run.smithi111:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-10-03T17:22:39.775 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-10-03T17:22:39.781 INFO:teuthology.task.internal:Enabling coredump saving... 2023-10-03T17:22:39.781 DEBUG:teuthology.orchestra.run.smithi111:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-10-03T17:22:39.837 INFO:teuthology.orchestra.run.smithi111.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-03T17:22:39.845 INFO:teuthology.orchestra.run.smithi111.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-03T17:22:39.846 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-10-03T17:22:39.853 INFO:teuthology.task.internal:Configuring sudo... 2023-10-03T17:22:39.854 DEBUG:teuthology.orchestra.run.smithi111:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-10-03T17:22:39.903 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-10-03T17:22:39.914 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-10-03T17:22:39.914 DEBUG:teuthology.orchestra.run.smithi111:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-10-03T17:22:39.951 DEBUG:teuthology.orchestra.run.smithi111:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-03T17:22:39.999 DEBUG:teuthology.orchestra.run.smithi111:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-03T17:22:40.047 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-10-03T17:22:40.048 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-10-03T17:22:40.129 DEBUG:teuthology.orchestra.run.smithi111:> sudo service rsyslog restart 2023-10-03T17:22:40.222 INFO:teuthology.run_tasks:Running task internal.timer... 2023-10-03T17:22:40.229 INFO:teuthology.task.internal:Starting timer... 2023-10-03T17:22:40.229 INFO:teuthology.run_tasks:Running task pcp... 2023-10-03T17:22:40.240 INFO:teuthology.run_tasks:Running task selinux... 2023-10-03T17:22:40.249 DEBUG:teuthology.task.selinux:Excluding smithi111: OS 'ubuntu' does not support SELinux 2023-10-03T17:22:40.249 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-10-03T17:22:40.249 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2023-10-03T17:22:40.249 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-10-03T17:22:40.250 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-10-03T17:22:40.258 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main was just updated or references a specific commit; assuming it is current 2023-10-03T17:22:40.258 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-10-03T17:22:40.279 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-10-03T17:22:40.281 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi111.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-10-03T17:32:24.579 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi111.front.sepia.ceph.com')] 2023-10-03T17:32:24.580 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi111.front.sepia.ceph.com' 2023-10-03T17:32:24.581 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi111.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-03T17:32:24.661 DEBUG:teuthology.orchestra.run.smithi111:> true 2023-10-03T17:32:24.774 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi111.front.sepia.ceph.com' 2023-10-03T17:32:24.774 INFO:teuthology.run_tasks:Running task clock... 2023-10-03T17:32:24.784 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-10-03T17:32:24.784 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-10-03T17:32:24.785 DEBUG:teuthology.orchestra.run.smithi111:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-03T17:32:24.843 INFO:teuthology.orchestra.run.smithi111.stdout: 3 Oct 17:32:24 ntpd[27835]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2023-10-03T17:32:24.843 INFO:teuthology.orchestra.run.smithi111.stdout: 3 Oct 17:32:24 ntpd[27835]: Command line: ntpd -gq 2023-10-03T17:32:24.843 INFO:teuthology.orchestra.run.smithi111.stdout: 3 Oct 17:32:24 ntpd[27835]: ---------------------------------------------------- 2023-10-03T17:32:24.844 INFO:teuthology.orchestra.run.smithi111.stdout: 3 Oct 17:32:24 ntpd[27835]: ntp-4 is maintained by Network Time Foundation, 2023-10-03T17:32:24.844 INFO:teuthology.orchestra.run.smithi111.stdout: 3 Oct 17:32:24 ntpd[27835]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2023-10-03T17:32:24.844 INFO:teuthology.orchestra.run.smithi111.stdout: 3 Oct 17:32:24 ntpd[27835]: corporation. Support and training for ntp-4 are 2023-10-03T17:32:24.844 INFO:teuthology.orchestra.run.smithi111.stdout: 3 Oct 17:32:24 ntpd[27835]: available at https://www.nwtime.org/support 2023-10-03T17:32:24.844 INFO:teuthology.orchestra.run.smithi111.stdout: 3 Oct 17:32:24 ntpd[27835]: ---------------------------------------------------- 2023-10-03T17:32:24.845 INFO:teuthology.orchestra.run.smithi111.stdout: 3 Oct 17:32:24 ntpd[27835]: proto: precision = 0.058 usec (-24) 2023-10-03T17:32:24.845 INFO:teuthology.orchestra.run.smithi111.stdout: 3 Oct 17:32:24 ntpd[27835]: basedate set to 2022-02-04 2023-10-03T17:32:24.845 INFO:teuthology.orchestra.run.smithi111.stdout: 3 Oct 17:32:24 ntpd[27835]: gps base set to 2022-02-06 (week 2196) 2023-10-03T17:32:24.845 INFO:teuthology.orchestra.run.smithi111.stdout: 3 Oct 17:32:24 ntpd[27835]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-10-03T17:32:24.845 INFO:teuthology.orchestra.run.smithi111.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-10-03T17:32:24.845 INFO:teuthology.orchestra.run.smithi111.stderr:restrict ::: KOD does nothing without LIMITED. 2023-10-03T17:32:24.846 INFO:teuthology.orchestra.run.smithi111.stdout: 3 Oct 17:32:24 ntpd[27835]: restrict ::: KOD does nothing without LIMITED. 2023-10-03T17:32:24.846 INFO:teuthology.orchestra.run.smithi111.stdout: 3 Oct 17:32:24 ntpd[27835]: Listen and drop on 0 v6wildcard [::]:123 2023-10-03T17:32:24.846 INFO:teuthology.orchestra.run.smithi111.stdout: 3 Oct 17:32:24 ntpd[27835]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2023-10-03T17:32:24.846 INFO:teuthology.orchestra.run.smithi111.stdout: 3 Oct 17:32:24 ntpd[27835]: Listen normally on 2 lo 127.0.0.1:123 2023-10-03T17:32:24.847 INFO:teuthology.orchestra.run.smithi111.stdout: 3 Oct 17:32:24 ntpd[27835]: Listen normally on 3 enp3s0f1 172.21.15.111:123 2023-10-03T17:32:24.847 INFO:teuthology.orchestra.run.smithi111.stdout: 3 Oct 17:32:24 ntpd[27835]: Listen normally on 4 lo [::1]:123 2023-10-03T17:32:24.847 INFO:teuthology.orchestra.run.smithi111.stdout: 3 Oct 17:32:24 ntpd[27835]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:cc2f%5]:123 2023-10-03T17:32:24.847 INFO:teuthology.orchestra.run.smithi111.stdout: 3 Oct 17:32:24 ntpd[27835]: Listening on routing socket on fd #22 for interface updates 2023-10-03T17:32:25.844 INFO:teuthology.orchestra.run.smithi111.stderr: 3 Oct 17:32:25 ntpd[27835]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-03T17:32:25.844 INFO:teuthology.orchestra.run.smithi111.stderr: 3 Oct 17:32:25 ntpd[27835]: can't open /var/log/ntpstats/rawstats.20231003: Permission denied 2023-10-03T17:32:25.844 INFO:teuthology.orchestra.run.smithi111.stderr: 3 Oct 17:32:25 ntpd[27835]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-03T17:32:25.844 INFO:teuthology.orchestra.run.smithi111.stderr: 3 Oct 17:32:25 ntpd[27835]: can't open /var/log/ntpstats/rawstats.20231003: Permission denied 2023-10-03T17:32:25.844 INFO:teuthology.orchestra.run.smithi111.stderr: 3 Oct 17:32:25 ntpd[27835]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-03T17:32:25.845 INFO:teuthology.orchestra.run.smithi111.stderr: 3 Oct 17:32:25 ntpd[27835]: can't open /var/log/ntpstats/rawstats.20231003: Permission denied 2023-10-03T17:32:25.845 INFO:teuthology.orchestra.run.smithi111.stderr: 3 Oct 17:32:25 ntpd[27835]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-03T17:32:25.845 INFO:teuthology.orchestra.run.smithi111.stderr: 3 Oct 17:32:25 ntpd[27835]: can't open /var/log/ntpstats/peerstats.20231003: Permission denied 2023-10-03T17:32:25.845 INFO:teuthology.orchestra.run.smithi111.stderr: 3 Oct 17:32:25 ntpd[27835]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-03T17:32:25.845 INFO:teuthology.orchestra.run.smithi111.stderr: 3 Oct 17:32:25 ntpd[27835]: can't open /var/log/ntpstats/rawstats.20231003: Permission denied 2023-10-03T17:32:25.845 INFO:teuthology.orchestra.run.smithi111.stderr: 3 Oct 17:32:25 ntpd[27835]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-03T17:32:25.845 INFO:teuthology.orchestra.run.smithi111.stderr: 3 Oct 17:32:25 ntpd[27835]: can't open /var/log/ntpstats/peerstats.20231003: Permission denied 2023-10-03T17:32:26.843 INFO:teuthology.orchestra.run.smithi111.stderr: 3 Oct 17:32:26 ntpd[27835]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-03T17:32:26.844 INFO:teuthology.orchestra.run.smithi111.stderr: 3 Oct 17:32:26 ntpd[27835]: can't open /var/log/ntpstats/rawstats.20231003: Permission denied 2023-10-03T17:32:27.847 INFO:teuthology.orchestra.run.smithi111.stderr: 3 Oct 17:32:27 ntpd[27835]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-03T17:32:27.848 INFO:teuthology.orchestra.run.smithi111.stderr: 3 Oct 17:32:27 ntpd[27835]: can't open /var/log/ntpstats/rawstats.20231003: Permission denied 2023-10-03T17:32:27.848 INFO:teuthology.orchestra.run.smithi111.stderr: 3 Oct 17:32:27 ntpd[27835]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-03T17:32:27.848 INFO:teuthology.orchestra.run.smithi111.stderr: 3 Oct 17:32:27 ntpd[27835]: can't open /var/log/ntpstats/rawstats.20231003: Permission denied 2023-10-03T17:32:27.848 INFO:teuthology.orchestra.run.smithi111.stderr: 3 Oct 17:32:27 ntpd[27835]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-03T17:32:27.848 INFO:teuthology.orchestra.run.smithi111.stderr: 3 Oct 17:32:27 ntpd[27835]: can't open /var/log/ntpstats/rawstats.20231003: Permission denied 2023-10-03T17:32:27.848 INFO:teuthology.orchestra.run.smithi111.stderr: 3 Oct 17:32:27 ntpd[27835]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-03T17:32:27.848 INFO:teuthology.orchestra.run.smithi111.stderr: 3 Oct 17:32:27 ntpd[27835]: can't open /var/log/ntpstats/peerstats.20231003: Permission denied 2023-10-03T17:32:27.848 INFO:teuthology.orchestra.run.smithi111.stderr: 3 Oct 17:32:27 ntpd[27835]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-03T17:32:27.848 INFO:teuthology.orchestra.run.smithi111.stderr: 3 Oct 17:32:27 ntpd[27835]: can't open /var/log/ntpstats/rawstats.20231003: Permission denied 2023-10-03T17:32:27.848 INFO:teuthology.orchestra.run.smithi111.stderr: 3 Oct 17:32:27 ntpd[27835]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-03T17:32:27.848 INFO:teuthology.orchestra.run.smithi111.stderr: 3 Oct 17:32:27 ntpd[27835]: can't open /var/log/ntpstats/peerstats.20231003: Permission denied 2023-10-03T17:32:29.846 INFO:teuthology.orchestra.run.smithi111.stderr: 3 Oct 17:32:29 ntpd[27835]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-03T17:32:29.846 INFO:teuthology.orchestra.run.smithi111.stderr: 3 Oct 17:32:29 ntpd[27835]: can't open /var/log/ntpstats/rawstats.20231003: Permission denied 2023-10-03T17:32:29.846 INFO:teuthology.orchestra.run.smithi111.stderr: 3 Oct 17:32:29 ntpd[27835]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-03T17:32:29.847 INFO:teuthology.orchestra.run.smithi111.stderr: 3 Oct 17:32:29 ntpd[27835]: can't open /var/log/ntpstats/rawstats.20231003: Permission denied 2023-10-03T17:32:29.847 INFO:teuthology.orchestra.run.smithi111.stderr: 3 Oct 17:32:29 ntpd[27835]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-03T17:32:29.847 INFO:teuthology.orchestra.run.smithi111.stderr: 3 Oct 17:32:29 ntpd[27835]: can't open /var/log/ntpstats/rawstats.20231003: Permission denied 2023-10-03T17:32:29.847 INFO:teuthology.orchestra.run.smithi111.stderr: 3 Oct 17:32:29 ntpd[27835]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-03T17:32:29.847 INFO:teuthology.orchestra.run.smithi111.stderr: 3 Oct 17:32:29 ntpd[27835]: can't open /var/log/ntpstats/rawstats.20231003: Permission denied 2023-10-03T17:32:29.847 INFO:teuthology.orchestra.run.smithi111.stderr: 3 Oct 17:32:29 ntpd[27835]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-03T17:32:29.848 INFO:teuthology.orchestra.run.smithi111.stderr: 3 Oct 17:32:29 ntpd[27835]: can't open /var/log/ntpstats/peerstats.20231003: Permission denied 2023-10-03T17:32:29.848 INFO:teuthology.orchestra.run.smithi111.stderr: 3 Oct 17:32:29 ntpd[27835]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-03T17:32:29.848 INFO:teuthology.orchestra.run.smithi111.stderr: 3 Oct 17:32:29 ntpd[27835]: can't open /var/log/ntpstats/rawstats.20231003: Permission denied 2023-10-03T17:32:29.848 INFO:teuthology.orchestra.run.smithi111.stderr: 3 Oct 17:32:29 ntpd[27835]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-03T17:32:29.848 INFO:teuthology.orchestra.run.smithi111.stderr: 3 Oct 17:32:29 ntpd[27835]: can't open /var/log/ntpstats/peerstats.20231003: Permission denied 2023-10-03T17:32:29.848 INFO:teuthology.orchestra.run.smithi111.stderr: 3 Oct 17:32:29 ntpd[27835]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-03T17:32:29.849 INFO:teuthology.orchestra.run.smithi111.stderr: 3 Oct 17:32:29 ntpd[27835]: can't open /var/log/ntpstats/rawstats.20231003: Permission denied 2023-10-03T17:32:29.849 INFO:teuthology.orchestra.run.smithi111.stderr: 3 Oct 17:32:29 ntpd[27835]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-03T17:32:29.849 INFO:teuthology.orchestra.run.smithi111.stderr: 3 Oct 17:32:29 ntpd[27835]: can't open /var/log/ntpstats/peerstats.20231003: Permission denied 2023-10-03T17:32:31.844 INFO:teuthology.orchestra.run.smithi111.stderr: 3 Oct 17:32:31 ntpd[27835]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-03T17:32:31.844 INFO:teuthology.orchestra.run.smithi111.stderr: 3 Oct 17:32:31 ntpd[27835]: can't open /var/log/ntpstats/rawstats.20231003: Permission denied 2023-10-03T17:32:31.844 INFO:teuthology.orchestra.run.smithi111.stderr: 3 Oct 17:32:31 ntpd[27835]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-03T17:32:31.845 INFO:teuthology.orchestra.run.smithi111.stderr: 3 Oct 17:32:31 ntpd[27835]: can't open /var/log/ntpstats/rawstats.20231003: Permission denied 2023-10-03T17:32:31.845 INFO:teuthology.orchestra.run.smithi111.stderr: 3 Oct 17:32:31 ntpd[27835]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-03T17:32:31.845 INFO:teuthology.orchestra.run.smithi111.stderr: 3 Oct 17:32:31 ntpd[27835]: can't open /var/log/ntpstats/rawstats.20231003: Permission denied 2023-10-03T17:32:31.845 INFO:teuthology.orchestra.run.smithi111.stderr: 3 Oct 17:32:31 ntpd[27835]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-03T17:32:31.845 INFO:teuthology.orchestra.run.smithi111.stderr: 3 Oct 17:32:31 ntpd[27835]: can't open /var/log/ntpstats/rawstats.20231003: Permission denied 2023-10-03T17:32:31.845 INFO:teuthology.orchestra.run.smithi111.stderr: 3 Oct 17:32:31 ntpd[27835]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-03T17:32:31.845 INFO:teuthology.orchestra.run.smithi111.stderr: 3 Oct 17:32:31 ntpd[27835]: can't open /var/log/ntpstats/peerstats.20231003: Permission denied 2023-10-03T17:32:31.845 INFO:teuthology.orchestra.run.smithi111.stderr: 3 Oct 17:32:31 ntpd[27835]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-03T17:32:31.845 INFO:teuthology.orchestra.run.smithi111.stderr: 3 Oct 17:32:31 ntpd[27835]: can't open /var/log/ntpstats/rawstats.20231003: Permission denied 2023-10-03T17:32:31.845 INFO:teuthology.orchestra.run.smithi111.stderr: 3 Oct 17:32:31 ntpd[27835]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-03T17:32:31.845 INFO:teuthology.orchestra.run.smithi111.stderr: 3 Oct 17:32:31 ntpd[27835]: can't open /var/log/ntpstats/peerstats.20231003: Permission denied 2023-10-03T17:32:31.846 INFO:teuthology.orchestra.run.smithi111.stdout: 3 Oct 17:32:31 ntpd[27835]: ntpd: time slew -0.001693 s 2023-10-03T17:32:31.846 INFO:teuthology.orchestra.run.smithi111.stdout:ntpd: time slew -0.001693s 2023-10-03T17:32:31.846 INFO:teuthology.orchestra.run.smithi111.stderr: 3 Oct 17:32:31 ntpd[27835]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2023-10-03T17:32:31.846 INFO:teuthology.orchestra.run.smithi111.stderr: 3 Oct 17:32:31 ntpd[27835]: can't open /var/log/ntpstats/loopstats.20231003: Permission denied 2023-10-03T17:32:31.947 INFO:teuthology.orchestra.run.smithi111.stdout: remote refid st t when poll reach delay offset jitter 2023-10-03T17:32:31.947 INFO:teuthology.orchestra.run.smithi111.stdout:============================================================================== 2023-10-03T17:32:31.947 INFO:teuthology.orchestra.run.smithi111.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-10-03T17:32:31.947 INFO:teuthology.orchestra.run.smithi111.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-10-03T17:32:31.947 INFO:teuthology.orchestra.run.smithi111.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-10-03T17:32:31.947 INFO:teuthology.orchestra.run.smithi111.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-10-03T17:32:31.948 INFO:teuthology.run_tasks:Running task install... 2023-10-03T17:32:31.957 DEBUG:teuthology.task.install:project ceph 2023-10-03T17:32:31.958 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'a7de91bfa5778a0d559a7e9bc8e35f93456b41ad'}} 2023-10-03T17:32:31.958 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': 'a7de91bfa5778a0d559a7e9bc8e35f93456b41ad'} 2023-10-03T17:32:31.958 INFO:teuthology.task.install:Using flavor: default 2023-10-03T17:32:31.966 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-10-03T17:32:31.966 INFO:teuthology.task.install:extra packages: [] 2023-10-03T17:32:31.966 DEBUG:teuthology.orchestra.run.smithi111:> sudo apt-key list | grep Ceph 2023-10-03T17:32:32.046 INFO:teuthology.orchestra.run.smithi111.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2023-10-03T17:32:32.075 INFO:teuthology.orchestra.run.smithi111.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2023-10-03T17:32:32.076 INFO:teuthology.orchestra.run.smithi111.stdout:uid [ unknown] Ceph.com (release key) 2023-10-03T17:32:32.076 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2023-10-03T17:32:32.076 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=a7de91bfa5778a0d559a7e9bc8e35f93456b41ad 2023-10-03T17:32:32.222 INFO:teuthology.task.install.deb:Pulling from https://4.chacra.ceph.com/r/ceph/wip-ajarr-63009-only-reacquire-kickstart/a7de91bfa5778a0d559a7e9bc8e35f93456b41ad/ubuntu/jammy/flavors/default/ 2023-10-03T17:32:32.222 INFO:teuthology.task.install.deb:Package version is 18.0.0-6336-ga7de91bf-1jammy 2023-10-03T17:32:32.525 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-10-03T17:32:32.525 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2023-10-03T17:32:32.540 DEBUG:teuthology.orchestra.run.smithi111:> sudo apt-get update 2023-10-03T17:32:32.742 INFO:teuthology.orchestra.run.smithi111.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2023-10-03T17:32:32.839 INFO:teuthology.orchestra.run.smithi111.stdout:Ign:2 https://4.chacra.ceph.com/r/ceph/wip-ajarr-63009-only-reacquire-kickstart/a7de91bfa5778a0d559a7e9bc8e35f93456b41ad/ubuntu/jammy/flavors/default jammy InRelease 2023-10-03T17:32:32.862 INFO:teuthology.orchestra.run.smithi111.stdout:Get:3 https://4.chacra.ceph.com/r/ceph/wip-ajarr-63009-only-reacquire-kickstart/a7de91bfa5778a0d559a7e9bc8e35f93456b41ad/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2023-10-03T17:32:32.862 INFO:teuthology.orchestra.run.smithi111.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy InRelease 2023-10-03T17:32:32.891 INFO:teuthology.orchestra.run.smithi111.stdout:Ign:5 https://4.chacra.ceph.com/r/ceph/wip-ajarr-63009-only-reacquire-kickstart/a7de91bfa5778a0d559a7e9bc8e35f93456b41ad/ubuntu/jammy/flavors/default jammy Release.gpg 2023-10-03T17:32:32.917 INFO:teuthology.orchestra.run.smithi111.stdout:Get:6 https://4.chacra.ceph.com/r/ceph/wip-ajarr-63009-only-reacquire-kickstart/a7de91bfa5778a0d559a7e9bc8e35f93456b41ad/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.6 kB] 2023-10-03T17:32:32.946 INFO:teuthology.orchestra.run.smithi111.stdout:Hit:7 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2023-10-03T17:32:32.973 INFO:teuthology.orchestra.run.smithi111.stdout:Get:8 https://4.chacra.ceph.com/r/ceph/wip-ajarr-63009-only-reacquire-kickstart/a7de91bfa5778a0d559a7e9bc8e35f93456b41ad/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,407 B] 2023-10-03T17:32:33.025 INFO:teuthology.orchestra.run.smithi111.stdout:Hit:9 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2023-10-03T17:32:33.158 INFO:teuthology.orchestra.run.smithi111.stdout:Fetched 28.6 kB in 0s (64.5 kB/s) 2023-10-03T17:32:34.041 INFO:teuthology.orchestra.run.smithi111.stdout:Reading package lists... 2023-10-03T17:32:34.056 DEBUG:teuthology.orchestra.run.smithi111:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=18.0.0-6336-ga7de91bf-1jammy cephadm=18.0.0-6336-ga7de91bf-1jammy ceph-mds=18.0.0-6336-ga7de91bf-1jammy ceph-mgr=18.0.0-6336-ga7de91bf-1jammy ceph-common=18.0.0-6336-ga7de91bf-1jammy ceph-fuse=18.0.0-6336-ga7de91bf-1jammy ceph-test=18.0.0-6336-ga7de91bf-1jammy ceph-volume=18.0.0-6336-ga7de91bf-1jammy radosgw=18.0.0-6336-ga7de91bf-1jammy python3-rados=18.0.0-6336-ga7de91bf-1jammy python3-rgw=18.0.0-6336-ga7de91bf-1jammy python3-cephfs=18.0.0-6336-ga7de91bf-1jammy python3-rbd=18.0.0-6336-ga7de91bf-1jammy libcephfs2=18.0.0-6336-ga7de91bf-1jammy libcephfs-dev=18.0.0-6336-ga7de91bf-1jammy librados2=18.0.0-6336-ga7de91bf-1jammy librbd1=18.0.0-6336-ga7de91bf-1jammy rbd-fuse=18.0.0-6336-ga7de91bf-1jammy 2023-10-03T17:32:34.107 INFO:teuthology.orchestra.run.smithi111.stdout:Reading package lists... 2023-10-03T17:32:34.270 INFO:teuthology.orchestra.run.smithi111.stdout:Building dependency tree... 2023-10-03T17:32:34.271 INFO:teuthology.orchestra.run.smithi111.stdout:Reading state information... 2023-10-03T17:32:34.414 INFO:teuthology.orchestra.run.smithi111.stdout:The following packages were automatically installed and are no longer required: 2023-10-03T17:32:34.415 INFO:teuthology.orchestra.run.smithi111.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-10-03T17:32:34.415 INFO:teuthology.orchestra.run.smithi111.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-10-03T17:32:34.415 INFO:teuthology.orchestra.run.smithi111.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-10-03T17:32:34.415 INFO:teuthology.orchestra.run.smithi111.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2023-10-03T17:32:34.415 INFO:teuthology.orchestra.run.smithi111.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2023-10-03T17:32:34.415 INFO:teuthology.orchestra.run.smithi111.stdout:Use 'sudo apt autoremove' to remove them. 2023-10-03T17:32:34.416 INFO:teuthology.orchestra.run.smithi111.stdout:The following additional packages will be installed: 2023-10-03T17:32:34.416 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2023-10-03T17:32:34.416 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2023-10-03T17:32:34.416 INFO:teuthology.orchestra.run.smithi111.stdout: libdouble-conversion3 libfmt8 libjq1 liblttng-ust1 liboath0 libonig5 2023-10-03T17:32:34.416 INFO:teuthology.orchestra.run.smithi111.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2023-10-03T17:32:34.416 INFO:teuthology.orchestra.run.smithi111.stdout: librdkafka1 librgw2 libsqlite3-mod-ceph libthrift-0.16.0 nvme-cli 2023-10-03T17:32:34.417 INFO:teuthology.orchestra.run.smithi111.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2023-10-03T17:32:34.417 INFO:teuthology.orchestra.run.smithi111.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2023-10-03T17:32:34.417 INFO:teuthology.orchestra.run.smithi111.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2023-10-03T17:32:34.417 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2023-10-03T17:32:34.417 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2023-10-03T17:32:34.417 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2023-10-03T17:32:34.417 INFO:teuthology.orchestra.run.smithi111.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2023-10-03T17:32:34.417 INFO:teuthology.orchestra.run.smithi111.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2023-10-03T17:32:34.417 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2023-10-03T17:32:34.417 INFO:teuthology.orchestra.run.smithi111.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2023-10-03T17:32:34.417 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2023-10-03T17:32:34.417 INFO:teuthology.orchestra.run.smithi111.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2023-10-03T17:32:34.418 INFO:teuthology.orchestra.run.smithi111.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2023-10-03T17:32:34.418 INFO:teuthology.orchestra.run.smithi111.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2023-10-03T17:32:34.418 INFO:teuthology.orchestra.run.smithi111.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2023-10-03T17:32:34.418 INFO:teuthology.orchestra.run.smithi111.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2023-10-03T17:32:34.418 INFO:teuthology.orchestra.run.smithi111.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat xmlstarlet 2023-10-03T17:32:34.418 INFO:teuthology.orchestra.run.smithi111.stdout:Suggested packages: 2023-10-03T17:32:34.418 INFO:teuthology.orchestra.run.smithi111.stdout: python3-influxdb python-jinja2-doc python3-beaker python-mako-doc 2023-10-03T17:32:34.418 INFO:teuthology.orchestra.run.smithi111.stdout: python-natsort-doc python-openssl-doc python3-openssl-dbg httpd-wsgi 2023-10-03T17:32:34.419 INFO:teuthology.orchestra.run.smithi111.stdout: libapache2-mod-python libapache2-mod-scgi libjs-mochikit python-pecan-doc 2023-10-03T17:32:34.419 INFO:teuthology.orchestra.run.smithi111.stdout: python-psutil-doc subversion python-pygments-doc python-pyinotify-doc 2023-10-03T17:32:34.419 INFO:teuthology.orchestra.run.smithi111.stdout: python3-socks python-requests-doc python3-dap python-sklearn-doc ipython3 2023-10-03T17:32:34.419 INFO:teuthology.orchestra.run.smithi111.stdout: python-waitress-doc python-webob-doc python-webtest-doc python-werkzeug-doc 2023-10-03T17:32:34.419 INFO:teuthology.orchestra.run.smithi111.stdout: python3-watchdog 2023-10-03T17:32:34.419 INFO:teuthology.orchestra.run.smithi111.stdout:Recommended packages: 2023-10-03T17:32:34.419 INFO:teuthology.orchestra.run.smithi111.stdout: btrfs-tools 2023-10-03T17:32:34.513 INFO:teuthology.orchestra.run.smithi111.stdout:The following NEW packages will be installed: 2023-10-03T17:32:34.513 INFO:teuthology.orchestra.run.smithi111.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2023-10-03T17:32:34.513 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2023-10-03T17:32:34.513 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2023-10-03T17:32:34.513 INFO:teuthology.orchestra.run.smithi111.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libfmt8 libjq1 liblttng-ust1 2023-10-03T17:32:34.513 INFO:teuthology.orchestra.run.smithi111.stdout: liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 2023-10-03T17:32:34.513 INFO:teuthology.orchestra.run.smithi111.stdout: librados2 libradosstriper1 librbd1 librdkafka1 librgw2 libsqlite3-mod-ceph 2023-10-03T17:32:34.513 INFO:teuthology.orchestra.run.smithi111.stdout: libthrift-0.16.0 nvme-cli python-asyncssh-doc python-babel-localedata 2023-10-03T17:32:34.514 INFO:teuthology.orchestra.run.smithi111.stdout: python-pastedeploy-tpl python3-asyncssh python3-babel python3-bcrypt 2023-10-03T17:32:34.514 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2023-10-03T17:32:34.514 INFO:teuthology.orchestra.run.smithi111.stdout: python3-certifi python3-cheroot python3-cherrypy3 python3-google-auth 2023-10-03T17:32:34.514 INFO:teuthology.orchestra.run.smithi111.stdout: python3-idna python3-iniconfig python3-jaraco.classes 2023-10-03T17:32:34.514 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jaraco.collections python3-jaraco.functools python3-jaraco.text 2023-10-03T17:32:34.514 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jinja2 python3-joblib python3-kubernetes python3-logutils 2023-10-03T17:32:34.514 INFO:teuthology.orchestra.run.smithi111.stdout: python3-mako python3-markupsafe python3-natsort python3-openssl 2023-10-03T17:32:34.514 INFO:teuthology.orchestra.run.smithi111.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2023-10-03T17:32:34.514 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pluggy python3-portend python3-prettytable python3-psutil python3-py 2023-10-03T17:32:34.514 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pyasn1 python3-pyasn1-modules python3-pygments python3-pyinotify 2023-10-03T17:32:34.514 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pytest python3-rados python3-rbd python3-repoze.lru python3-requests 2023-10-03T17:32:34.515 INFO:teuthology.orchestra.run.smithi111.stdout: python3-requests-oauthlib python3-rgw python3-routes python3-rsa 2023-10-03T17:32:34.515 INFO:teuthology.orchestra.run.smithi111.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2023-10-03T17:32:34.515 INFO:teuthology.orchestra.run.smithi111.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2023-10-03T17:32:34.515 INFO:teuthology.orchestra.run.smithi111.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2023-10-03T17:32:34.515 INFO:teuthology.orchestra.run.smithi111.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2023-10-03T17:32:34.515 INFO:teuthology.orchestra.run.smithi111.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n radosgw rbd-fuse 2023-10-03T17:32:34.515 INFO:teuthology.orchestra.run.smithi111.stdout: socat xmlstarlet 2023-10-03T17:32:34.698 INFO:teuthology.orchestra.run.smithi111.stdout:0 upgraded, 109 newly installed, 0 to remove and 254 not upgraded. 2023-10-03T17:32:34.698 INFO:teuthology.orchestra.run.smithi111.stdout:Need to get 183 MB of archives. 2023-10-03T17:32:34.698 INFO:teuthology.orchestra.run.smithi111.stdout:After this operation, 823 MB of additional disk space will be used. 2023-10-03T17:32:34.698 INFO:teuthology.orchestra.run.smithi111.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libfmt8 amd64 8.1.1+ds1-2 [60.2 kB] 2023-10-03T17:32:34.739 INFO:teuthology.orchestra.run.smithi111.stdout:Get:2 https://4.chacra.ceph.com/r/ceph/wip-ajarr-63009-only-reacquire-kickstart/a7de91bfa5778a0d559a7e9bc8e35f93456b41ad/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 18.0.0-6336-ga7de91bf-1jammy [3,437 kB] 2023-10-03T17:32:34.972 INFO:teuthology.orchestra.run.smithi111.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2023-10-03T17:32:35.112 INFO:teuthology.orchestra.run.smithi111.stdout:Get:4 https://4.chacra.ceph.com/r/ceph/wip-ajarr-63009-only-reacquire-kickstart/a7de91bfa5778a0d559a7e9bc8e35f93456b41ad/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 18.0.0-6336-ga7de91bf-1jammy [3,121 kB] 2023-10-03T17:32:35.142 INFO:teuthology.orchestra.run.smithi111.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2023-10-03T17:32:35.158 INFO:teuthology.orchestra.run.smithi111.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2023-10-03T17:32:35.228 INFO:teuthology.orchestra.run.smithi111.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2023-10-03T17:32:35.257 INFO:teuthology.orchestra.run.smithi111.stdout:Get:8 https://4.chacra.ceph.com/r/ceph/wip-ajarr-63009-only-reacquire-kickstart/a7de91bfa5778a0d559a7e9bc8e35f93456b41ad/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 18.0.0-6336-ga7de91bf-1jammy [676 kB] 2023-10-03T17:32:35.277 INFO:teuthology.orchestra.run.smithi111.stdout:Get:9 https://4.chacra.ceph.com/r/ceph/wip-ajarr-63009-only-reacquire-kickstart/a7de91bfa5778a0d559a7e9bc8e35f93456b41ad/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 18.0.0-6336-ga7de91bf-1jammy [345 kB] 2023-10-03T17:32:35.297 INFO:teuthology.orchestra.run.smithi111.stdout:Get:10 https://4.chacra.ceph.com/r/ceph/wip-ajarr-63009-only-reacquire-kickstart/a7de91bfa5778a0d559a7e9bc8e35f93456b41ad/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 18.0.0-6336-ga7de91bf-1jammy [23.0 kB] 2023-10-03T17:32:35.298 INFO:teuthology.orchestra.run.smithi111.stdout:Get:11 https://4.chacra.ceph.com/r/ceph/wip-ajarr-63009-only-reacquire-kickstart/a7de91bfa5778a0d559a7e9bc8e35f93456b41ad/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 18.0.0-6336-ga7de91bf-1jammy [170 kB] 2023-10-03T17:32:35.304 INFO:teuthology.orchestra.run.smithi111.stdout:Get:12 https://4.chacra.ceph.com/r/ceph/wip-ajarr-63009-only-reacquire-kickstart/a7de91bfa5778a0d559a7e9bc8e35f93456b41ad/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 18.0.0-6336-ga7de91bf-1jammy [52.4 kB] 2023-10-03T17:32:35.306 INFO:teuthology.orchestra.run.smithi111.stdout:Get:13 https://4.chacra.ceph.com/r/ceph/wip-ajarr-63009-only-reacquire-kickstart/a7de91bfa5778a0d559a7e9bc8e35f93456b41ad/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 18.0.0-6336-ga7de91bf-1jammy [318 kB] 2023-10-03T17:32:35.313 INFO:teuthology.orchestra.run.smithi111.stdout:Get:14 https://4.chacra.ceph.com/r/ceph/wip-ajarr-63009-only-reacquire-kickstart/a7de91bfa5778a0d559a7e9bc8e35f93456b41ad/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 18.0.0-6336-ga7de91bf-1jammy [6,201 kB] 2023-10-03T17:32:35.422 INFO:teuthology.orchestra.run.smithi111.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2023-10-03T17:32:35.433 INFO:teuthology.orchestra.run.smithi111.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2023-10-03T17:32:35.461 INFO:teuthology.orchestra.run.smithi111.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2023-10-03T17:32:35.470 INFO:teuthology.orchestra.run.smithi111.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2023-10-03T17:32:35.470 INFO:teuthology.orchestra.run.smithi111.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2023-10-03T17:32:35.472 INFO:teuthology.orchestra.run.smithi111.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2023-10-03T17:32:35.536 INFO:teuthology.orchestra.run.smithi111.stdout:Get:21 https://4.chacra.ceph.com/r/ceph/wip-ajarr-63009-only-reacquire-kickstart/a7de91bfa5778a0d559a7e9bc8e35f93456b41ad/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 18.0.0-6336-ga7de91bf-1jammy [102 kB] 2023-10-03T17:32:35.538 INFO:teuthology.orchestra.run.smithi111.stdout:Get:22 https://4.chacra.ceph.com/r/ceph/wip-ajarr-63009-only-reacquire-kickstart/a7de91bfa5778a0d559a7e9bc8e35f93456b41ad/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 18.0.0-6336-ga7de91bf-1jammy [428 kB] 2023-10-03T17:32:35.551 INFO:teuthology.orchestra.run.smithi111.stdout:Get:23 https://4.chacra.ceph.com/r/ceph/wip-ajarr-63009-only-reacquire-kickstart/a7de91bfa5778a0d559a7e9bc8e35f93456b41ad/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 18.0.0-6336-ga7de91bf-1jammy [22.5 MB] 2023-10-03T17:32:35.653 INFO:teuthology.orchestra.run.smithi111.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2023-10-03T17:32:35.869 INFO:teuthology.orchestra.run.smithi111.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2023-10-03T17:32:35.883 INFO:teuthology.orchestra.run.smithi111.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2023-10-03T17:32:36.003 INFO:teuthology.orchestra.run.smithi111.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2023-10-03T17:32:36.012 INFO:teuthology.orchestra.run.smithi111.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2023-10-03T17:32:36.023 INFO:teuthology.orchestra.run.smithi111.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2023-10-03T17:32:36.039 INFO:teuthology.orchestra.run.smithi111.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2023-10-03T17:32:36.053 INFO:teuthology.orchestra.run.smithi111.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2023-10-03T17:32:36.058 INFO:teuthology.orchestra.run.smithi111.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2023-10-03T17:32:36.062 INFO:teuthology.orchestra.run.smithi111.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2023-10-03T17:32:36.166 INFO:teuthology.orchestra.run.smithi111.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2023-10-03T17:32:36.340 INFO:teuthology.orchestra.run.smithi111.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2023-10-03T17:32:36.465 INFO:teuthology.orchestra.run.smithi111.stdout:Get:36 https://4.chacra.ceph.com/r/ceph/wip-ajarr-63009-only-reacquire-kickstart/a7de91bfa5778a0d559a7e9bc8e35f93456b41ad/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 18.0.0-6336-ga7de91bf-1jammy [6,840 kB] 2023-10-03T17:32:36.529 INFO:teuthology.orchestra.run.smithi111.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2023-10-03T17:32:36.550 INFO:teuthology.orchestra.run.smithi111.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2023-10-03T17:32:36.628 INFO:teuthology.orchestra.run.smithi111.stdout:Get:39 https://4.chacra.ceph.com/r/ceph/wip-ajarr-63009-only-reacquire-kickstart/a7de91bfa5778a0d559a7e9bc8e35f93456b41ad/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 18.0.0-6336-ga7de91bf-1jammy [228 kB] 2023-10-03T17:32:36.633 INFO:teuthology.orchestra.run.smithi111.stdout:Get:40 https://4.chacra.ceph.com/r/ceph/wip-ajarr-63009-only-reacquire-kickstart/a7de91bfa5778a0d559a7e9bc8e35f93456b41ad/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 18.0.0-6336-ga7de91bf-1jammy [117 kB] 2023-10-03T17:32:36.636 INFO:teuthology.orchestra.run.smithi111.stdout:Get:41 https://4.chacra.ceph.com/r/ceph/wip-ajarr-63009-only-reacquire-kickstart/a7de91bfa5778a0d559a7e9bc8e35f93456b41ad/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 18.0.0-6336-ga7de91bf-1jammy [1,487 kB] 2023-10-03T17:32:36.684 INFO:teuthology.orchestra.run.smithi111.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2023-10-03T17:32:36.701 INFO:teuthology.orchestra.run.smithi111.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2023-10-03T17:32:36.708 INFO:teuthology.orchestra.run.smithi111.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2023-10-03T17:32:36.748 INFO:teuthology.orchestra.run.smithi111.stdout:Get:45 https://4.chacra.ceph.com/r/ceph/wip-ajarr-63009-only-reacquire-kickstart/a7de91bfa5778a0d559a7e9bc8e35f93456b41ad/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 18.0.0-6336-ga7de91bf-1jammy [9,851 kB] 2023-10-03T17:32:36.781 INFO:teuthology.orchestra.run.smithi111.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2023-10-03T17:32:36.808 INFO:teuthology.orchestra.run.smithi111.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2023-10-03T17:32:36.816 INFO:teuthology.orchestra.run.smithi111.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2023-10-03T17:32:36.938 INFO:teuthology.orchestra.run.smithi111.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2023-10-03T17:32:36.939 INFO:teuthology.orchestra.run.smithi111.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2023-10-03T17:32:37.054 INFO:teuthology.orchestra.run.smithi111.stdout:Get:51 https://4.chacra.ceph.com/r/ceph/wip-ajarr-63009-only-reacquire-kickstart/a7de91bfa5778a0d559a7e9bc8e35f93456b41ad/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 18.0.0-6336-ga7de91bf-1jammy [31.7 MB] 2023-10-03T17:32:37.107 INFO:teuthology.orchestra.run.smithi111.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2023-10-03T17:32:37.296 INFO:teuthology.orchestra.run.smithi111.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2023-10-03T17:32:37.453 INFO:teuthology.orchestra.run.smithi111.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2023-10-03T17:32:37.558 INFO:teuthology.orchestra.run.smithi111.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2023-10-03T17:32:37.581 INFO:teuthology.orchestra.run.smithi111.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-urllib3 all 1.26.5-1~exp1 [96.4 kB] 2023-10-03T17:32:37.612 INFO:teuthology.orchestra.run.smithi111.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2023-10-03T17:32:37.624 INFO:teuthology.orchestra.run.smithi111.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2023-10-03T17:32:37.665 INFO:teuthology.orchestra.run.smithi111.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2023-10-03T17:32:37.674 INFO:teuthology.orchestra.run.smithi111.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2023-10-03T17:32:37.703 INFO:teuthology.orchestra.run.smithi111.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2023-10-03T17:32:37.937 INFO:teuthology.orchestra.run.smithi111.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2023-10-03T17:32:38.103 INFO:teuthology.orchestra.run.smithi111.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jinja2 all 3.0.3-1 [108 kB] 2023-10-03T17:32:38.398 INFO:teuthology.orchestra.run.smithi111.stdout:Get:64 https://4.chacra.ceph.com/r/ceph/wip-ajarr-63009-only-reacquire-kickstart/a7de91bfa5778a0d559a7e9bc8e35f93456b41ad/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 18.0.0-6336-ga7de91bf-1jammy [3,596 B] 2023-10-03T17:32:38.399 INFO:teuthology.orchestra.run.smithi111.stdout:Get:65 https://4.chacra.ceph.com/r/ceph/wip-ajarr-63009-only-reacquire-kickstart/a7de91bfa5778a0d559a7e9bc8e35f93456b41ad/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 18.0.0-6336-ga7de91bf-1jammy [827 kB] 2023-10-03T17:32:38.415 INFO:teuthology.orchestra.run.smithi111.stdout:Get:66 https://4.chacra.ceph.com/r/ceph/wip-ajarr-63009-only-reacquire-kickstart/a7de91bfa5778a0d559a7e9bc8e35f93456b41ad/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 18.0.0-6336-ga7de91bf-1jammy [2,135 kB] 2023-10-03T17:32:38.453 INFO:teuthology.orchestra.run.smithi111.stdout:Get:67 https://4.chacra.ceph.com/r/ceph/wip-ajarr-63009-only-reacquire-kickstart/a7de91bfa5778a0d559a7e9bc8e35f93456b41ad/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 18.0.0-6336-ga7de91bf-1jammy [251 kB] 2023-10-03T17:32:38.457 INFO:teuthology.orchestra.run.smithi111.stdout:Get:68 https://4.chacra.ceph.com/r/ceph/wip-ajarr-63009-only-reacquire-kickstart/a7de91bfa5778a0d559a7e9bc8e35f93456b41ad/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 18.0.0-6336-ga7de91bf-1jammy [128 kB] 2023-10-03T17:32:38.458 INFO:teuthology.orchestra.run.smithi111.stdout:Get:69 https://4.chacra.ceph.com/r/ceph/wip-ajarr-63009-only-reacquire-kickstart/a7de91bfa5778a0d559a7e9bc8e35f93456b41ad/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 18.0.0-6336-ga7de91bf-1jammy [1,883 kB] 2023-10-03T17:32:38.481 INFO:teuthology.orchestra.run.smithi111.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2023-10-03T17:32:38.483 INFO:teuthology.orchestra.run.smithi111.stdout:Get:71 https://4.chacra.ceph.com/r/ceph/wip-ajarr-63009-only-reacquire-kickstart/a7de91bfa5778a0d559a7e9bc8e35f93456b41ad/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 18.0.0-6336-ga7de91bf-1jammy [8,615 kB] 2023-10-03T17:32:38.500 INFO:teuthology.orchestra.run.smithi111.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2023-10-03T17:32:38.506 INFO:teuthology.orchestra.run.smithi111.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2023-10-03T17:32:38.570 INFO:teuthology.orchestra.run.smithi111.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2023-10-03T17:32:38.815 INFO:teuthology.orchestra.run.smithi111.stdout:Get:75 https://4.chacra.ceph.com/r/ceph/wip-ajarr-63009-only-reacquire-kickstart/a7de91bfa5778a0d559a7e9bc8e35f93456b41ad/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 18.0.0-6336-ga7de91bf-1jammy [3,722 B] 2023-10-03T17:32:38.815 INFO:teuthology.orchestra.run.smithi111.stdout:Get:76 https://4.chacra.ceph.com/r/ceph/wip-ajarr-63009-only-reacquire-kickstart/a7de91bfa5778a0d559a7e9bc8e35f93456b41ad/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 18.0.0-6336-ga7de91bf-1jammy [47.7 MB] 2023-10-03T17:32:38.866 INFO:teuthology.orchestra.run.smithi111.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2023-10-03T17:32:38.876 INFO:teuthology.orchestra.run.smithi111.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2023-10-03T17:32:38.877 INFO:teuthology.orchestra.run.smithi111.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2023-10-03T17:32:38.945 INFO:teuthology.orchestra.run.smithi111.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2023-10-03T17:32:38.946 INFO:teuthology.orchestra.run.smithi111.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2023-10-03T17:32:38.948 INFO:teuthology.orchestra.run.smithi111.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2023-10-03T17:32:39.113 INFO:teuthology.orchestra.run.smithi111.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2023-10-03T17:32:39.303 INFO:teuthology.orchestra.run.smithi111.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2023-10-03T17:32:39.366 INFO:teuthology.orchestra.run.smithi111.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2023-10-03T17:32:39.401 INFO:teuthology.orchestra.run.smithi111.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2023-10-03T17:32:39.460 INFO:teuthology.orchestra.run.smithi111.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2023-10-03T17:32:39.631 INFO:teuthology.orchestra.run.smithi111.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2023-10-03T17:32:39.669 INFO:teuthology.orchestra.run.smithi111.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2023-10-03T17:32:39.693 INFO:teuthology.orchestra.run.smithi111.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2023-10-03T17:32:39.701 INFO:teuthology.orchestra.run.smithi111.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2023-10-03T17:32:39.742 INFO:teuthology.orchestra.run.smithi111.stdout:Get:92 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2023-10-03T17:32:39.765 INFO:teuthology.orchestra.run.smithi111.stdout:Get:93 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2023-10-03T17:32:39.963 INFO:teuthology.orchestra.run.smithi111.stdout:Get:94 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2023-10-03T17:32:40.447 INFO:teuthology.orchestra.run.smithi111.stdout:Get:95 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2023-10-03T17:32:40.448 INFO:teuthology.orchestra.run.smithi111.stdout:Get:96 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2023-10-03T17:32:40.459 INFO:teuthology.orchestra.run.smithi111.stdout:Get:97 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2023-10-03T17:32:40.462 INFO:teuthology.orchestra.run.smithi111.stdout:Get:98 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2023-10-03T17:32:40.492 INFO:teuthology.orchestra.run.smithi111.stdout:Get:99 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2023-10-03T17:32:40.506 INFO:teuthology.orchestra.run.smithi111.stdout:Get:100 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2023-10-03T17:32:40.633 INFO:teuthology.orchestra.run.smithi111.stdout:Get:101 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2023-10-03T17:32:40.634 INFO:teuthology.orchestra.run.smithi111.stdout:Get:102 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2023-10-03T17:32:40.636 INFO:teuthology.orchestra.run.smithi111.stdout:Get:103 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2023-10-03T17:32:40.654 INFO:teuthology.orchestra.run.smithi111.stdout:Get:104 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2023-10-03T17:32:40.823 INFO:teuthology.orchestra.run.smithi111.stdout:Get:105 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2023-10-03T17:32:41.256 INFO:teuthology.orchestra.run.smithi111.stdout:Get:106 https://4.chacra.ceph.com/r/ceph/wip-ajarr-63009-only-reacquire-kickstart/a7de91bfa5778a0d559a7e9bc8e35f93456b41ad/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 18.0.0-6336-ga7de91bf-1jammy [234 kB] 2023-10-03T17:32:41.330 INFO:teuthology.orchestra.run.smithi111.stdout:Get:107 https://4.chacra.ceph.com/r/ceph/wip-ajarr-63009-only-reacquire-kickstart/a7de91bfa5778a0d559a7e9bc8e35f93456b41ad/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 18.0.0-6336-ga7de91bf-1jammy [28.3 kB] 2023-10-03T17:32:41.330 INFO:teuthology.orchestra.run.smithi111.stdout:Get:108 https://4.chacra.ceph.com/r/ceph/wip-ajarr-63009-only-reacquire-kickstart/a7de91bfa5778a0d559a7e9bc8e35f93456b41ad/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 18.0.0-6336-ga7de91bf-1jammy [12.3 MB] 2023-10-03T17:32:41.817 INFO:teuthology.orchestra.run.smithi111.stdout:Get:109 https://4.chacra.ceph.com/r/ceph/wip-ajarr-63009-only-reacquire-kickstart/a7de91bfa5778a0d559a7e9bc8e35f93456b41ad/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 18.0.0-6336-ga7de91bf-1jammy [80.9 kB] 2023-10-03T17:32:42.386 INFO:teuthology.orchestra.run.smithi111.stdout:Fetched 183 MB in 7s (25.1 MB/s) 2023-10-03T17:32:42.514 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package libfmt8:amd64. 2023-10-03T17:32:42.611 INFO:teuthology.orchestra.run.smithi111.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 166009 files and directories currently installed.) 2023-10-03T17:32:42.616 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../000-libfmt8_8.1.1+ds1-2_amd64.deb ... 2023-10-03T17:32:42.704 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking libfmt8:amd64 (8.1.1+ds1-2) ... 2023-10-03T17:32:43.007 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2023-10-03T17:32:43.033 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../001-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2023-10-03T17:32:43.065 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2023-10-03T17:32:43.377 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2023-10-03T17:32:43.404 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../002-libdouble-conversion3_3.1.7-4_amd64.deb ... 2023-10-03T17:32:43.444 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2023-10-03T17:32:43.780 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2023-10-03T17:32:43.806 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../003-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2023-10-03T17:32:43.846 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2023-10-03T17:32:44.292 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package libqt5core5a:amd64. 2023-10-03T17:32:44.319 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../004-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-10-03T17:32:44.366 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-03T17:32:44.753 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2023-10-03T17:32:44.779 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../005-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-10-03T17:32:44.820 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-03T17:32:45.164 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package libqt5network5:amd64. 2023-10-03T17:32:45.191 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../006-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-10-03T17:32:45.231 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-03T17:32:45.601 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2023-10-03T17:32:45.627 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../007-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2023-10-03T17:32:45.667 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2023-10-03T17:32:45.953 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package librados2. 2023-10-03T17:32:45.980 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../008-librados2_18.0.0-6336-ga7de91bf-1jammy_amd64.deb ... 2023-10-03T17:32:46.037 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking librados2 (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-03T17:32:46.489 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package librbd1. 2023-10-03T17:32:46.516 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../009-librbd1_18.0.0-6336-ga7de91bf-1jammy_amd64.deb ... 2023-10-03T17:32:46.556 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking librbd1 (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-03T17:32:46.992 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package libcephfs2. 2023-10-03T17:32:47.018 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../010-libcephfs2_18.0.0-6336-ga7de91bf-1jammy_amd64.deb ... 2023-10-03T17:32:47.058 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking libcephfs2 (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-03T17:32:47.319 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-rados. 2023-10-03T17:32:47.335 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../011-python3-rados_18.0.0-6336-ga7de91bf-1jammy_amd64.deb ... 2023-10-03T17:32:47.369 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-rados (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-03T17:32:47.681 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-ceph-argparse. 2023-10-03T17:32:47.707 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../012-python3-ceph-argparse_18.0.0-6336-ga7de91bf-1jammy_all.deb ... 2023-10-03T17:32:47.747 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-ceph-argparse (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-03T17:32:47.999 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-cephfs. 2023-10-03T17:32:48.014 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../013-python3-cephfs_18.0.0-6336-ga7de91bf-1jammy_amd64.deb ... 2023-10-03T17:32:48.058 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-cephfs (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-03T17:32:48.369 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-ceph-common. 2023-10-03T17:32:48.395 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../014-python3-ceph-common_18.0.0-6336-ga7de91bf-1jammy_all.deb ... 2023-10-03T17:32:48.435 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-ceph-common (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-03T17:32:48.738 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-wcwidth. 2023-10-03T17:32:48.765 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../015-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2023-10-03T17:32:48.797 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2023-10-03T17:32:49.091 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-prettytable. 2023-10-03T17:32:49.117 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../016-python3-prettytable_2.5.0-2_all.deb ... 2023-10-03T17:32:49.158 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2023-10-03T17:32:49.419 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-rbd. 2023-10-03T17:32:49.445 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../017-python3-rbd_18.0.0-6336-ga7de91bf-1jammy_amd64.deb ... 2023-10-03T17:32:49.485 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-rbd (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-03T17:32:49.830 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package librdkafka1:amd64. 2023-10-03T17:32:49.856 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../018-librdkafka1_1.8.0-1build1_amd64.deb ... 2023-10-03T17:32:49.913 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2023-10-03T17:32:50.249 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package librgw2. 2023-10-03T17:32:50.276 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../019-librgw2_18.0.0-6336-ga7de91bf-1jammy_amd64.deb ... 2023-10-03T17:32:50.316 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking librgw2 (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-03T17:32:50.818 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-rgw. 2023-10-03T17:32:50.833 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../020-python3-rgw_18.0.0-6336-ga7de91bf-1jammy_amd64.deb ... 2023-10-03T17:32:50.868 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-rgw (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-03T17:32:51.196 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package liboath0:amd64. 2023-10-03T17:32:51.222 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../021-liboath0_2.6.7-3build1_amd64.deb ... 2023-10-03T17:32:51.262 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2023-10-03T17:32:51.574 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package libradosstriper1. 2023-10-03T17:32:51.600 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../022-libradosstriper1_18.0.0-6336-ga7de91bf-1jammy_amd64.deb ... 2023-10-03T17:32:51.640 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking libradosstriper1 (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-03T17:32:52.010 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package ceph-common. 2023-10-03T17:32:52.036 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../023-ceph-common_18.0.0-6336-ga7de91bf-1jammy_amd64.deb ... 2023-10-03T17:32:52.085 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking ceph-common (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-03T17:32:53.624 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package ceph-base. 2023-10-03T17:32:53.651 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../024-ceph-base_18.0.0-6336-ga7de91bf-1jammy_amd64.deb ... 2023-10-03T17:32:53.701 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking ceph-base (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-03T17:32:54.412 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-jaraco.functools. 2023-10-03T17:32:54.438 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../025-python3-jaraco.functools_3.4.0-2_all.deb ... 2023-10-03T17:32:54.478 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2023-10-03T17:32:54.765 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-cheroot. 2023-10-03T17:32:54.791 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../026-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2023-10-03T17:32:54.831 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2023-10-03T17:32:55.176 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-jaraco.classes. 2023-10-03T17:32:55.202 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../027-python3-jaraco.classes_3.2.1-3_all.deb ... 2023-10-03T17:32:55.242 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2023-10-03T17:32:55.512 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-jaraco.text. 2023-10-03T17:32:55.538 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../028-python3-jaraco.text_3.6.0-2_all.deb ... 2023-10-03T17:32:55.570 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2023-10-03T17:32:55.840 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-jaraco.collections. 2023-10-03T17:32:55.866 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../029-python3-jaraco.collections_3.4.0-2_all.deb ... 2023-10-03T17:32:55.906 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2023-10-03T17:32:56.209 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-tempora. 2023-10-03T17:32:56.235 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../030-python3-tempora_4.1.2-1_all.deb ... 2023-10-03T17:32:56.275 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-tempora (4.1.2-1) ... 2023-10-03T17:32:56.587 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-portend. 2023-10-03T17:32:56.613 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../031-python3-portend_3.0.0-1_all.deb ... 2023-10-03T17:32:56.653 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-portend (3.0.0-1) ... 2023-10-03T17:32:56.914 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-zc.lockfile. 2023-10-03T17:32:56.930 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../032-python3-zc.lockfile_2.0-1_all.deb ... 2023-10-03T17:32:56.964 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2023-10-03T17:32:57.267 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-cherrypy3. 2023-10-03T17:32:57.293 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../033-python3-cherrypy3_18.6.1-4_all.deb ... 2023-10-03T17:32:57.334 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2023-10-03T17:32:57.687 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-natsort. 2023-10-03T17:32:57.714 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../034-python3-natsort_8.0.2-1_all.deb ... 2023-10-03T17:32:57.745 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-natsort (8.0.2-1) ... 2023-10-03T17:32:58.332 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-logutils. 2023-10-03T17:32:58.358 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../035-python3-logutils_0.3.3-8_all.deb ... 2023-10-03T17:32:58.398 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-logutils (0.3.3-8) ... 2023-10-03T17:32:58.676 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-markupsafe. 2023-10-03T17:32:58.691 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../036-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2023-10-03T17:32:58.727 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2023-10-03T17:32:58.987 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-mako. 2023-10-03T17:32:59.002 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../037-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2023-10-03T17:32:59.037 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2023-10-03T17:32:59.340 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-simplegeneric. 2023-10-03T17:32:59.366 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../038-python3-simplegeneric_0.8.1-3_all.deb ... 2023-10-03T17:32:59.406 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2023-10-03T17:32:59.709 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-singledispatch. 2023-10-03T17:32:59.736 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../039-python3-singledispatch_3.4.0.3-3_all.deb ... 2023-10-03T17:32:59.776 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2023-10-03T17:33:00.087 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-webob. 2023-10-03T17:33:00.114 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../040-python3-webob_1%3a1.8.6-1.1_all.deb ... 2023-10-03T17:33:00.154 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2023-10-03T17:33:00.465 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-waitress. 2023-10-03T17:33:00.480 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../041-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2023-10-03T17:33:00.518 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2023-10-03T17:33:00.793 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-tempita. 2023-10-03T17:33:00.819 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../042-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2023-10-03T17:33:00.859 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2023-10-03T17:33:01.162 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-paste. 2023-10-03T17:33:01.189 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../043-python3-paste_3.5.0+dfsg1-1_all.deb ... 2023-10-03T17:33:01.229 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2023-10-03T17:33:01.557 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2023-10-03T17:33:01.572 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../044-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2023-10-03T17:33:01.615 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2023-10-03T17:33:01.918 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-pastedeploy. 2023-10-03T17:33:01.944 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../045-python3-pastedeploy_2.1.1-1_all.deb ... 2023-10-03T17:33:01.984 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2023-10-03T17:33:02.288 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-webtest. 2023-10-03T17:33:02.314 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../046-python3-webtest_2.0.35-1_all.deb ... 2023-10-03T17:33:02.354 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-webtest (2.0.35-1) ... 2023-10-03T17:33:02.668 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-pecan. 2023-10-03T17:33:02.695 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../047-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2023-10-03T17:33:02.731 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2023-10-03T17:33:03.078 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-certifi. 2023-10-03T17:33:03.094 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../048-python3-certifi_2020.6.20-1_all.deb ... 2023-10-03T17:33:03.128 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2023-10-03T17:33:03.431 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-idna. 2023-10-03T17:33:03.457 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../049-python3-idna_3.3-1_all.deb ... 2023-10-03T17:33:03.497 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-idna (3.3-1) ... 2023-10-03T17:33:03.808 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-urllib3. 2023-10-03T17:33:03.835 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../050-python3-urllib3_1.26.5-1~exp1_all.deb ... 2023-10-03T17:33:03.875 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1) ... 2023-10-03T17:33:04.195 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-requests. 2023-10-03T17:33:04.222 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../051-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2023-10-03T17:33:04.261 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2023-10-03T17:33:04.556 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-werkzeug. 2023-10-03T17:33:04.583 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../052-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2023-10-03T17:33:04.622 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2023-10-03T17:33:04.925 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2023-10-03T17:33:04.952 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../053-ceph-mgr-modules-core_18.0.0-6336-ga7de91bf-1jammy_all.deb ... 2023-10-03T17:33:04.992 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking ceph-mgr-modules-core (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-03T17:33:05.364 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2023-10-03T17:33:05.390 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../054-libsqlite3-mod-ceph_18.0.0-6336-ga7de91bf-1jammy_amd64.deb ... 2023-10-03T17:33:05.422 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking libsqlite3-mod-ceph (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-03T17:33:05.741 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-openssl. 2023-10-03T17:33:05.768 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../055-python3-openssl_21.0.0-1_all.deb ... 2023-10-03T17:33:05.807 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-openssl (21.0.0-1) ... 2023-10-03T17:33:06.152 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package ceph-mgr. 2023-10-03T17:33:06.179 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../056-ceph-mgr_18.0.0-6336-ga7de91bf-1jammy_amd64.deb ... 2023-10-03T17:33:06.218 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking ceph-mgr (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-03T17:33:06.563 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package ceph-mon. 2023-10-03T17:33:06.579 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../057-ceph-mon_18.0.0-6336-ga7de91bf-1jammy_amd64.deb ... 2023-10-03T17:33:06.613 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking ceph-mon (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-03T17:33:07.467 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package ceph-osd. 2023-10-03T17:33:07.493 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../058-ceph-osd_18.0.0-6336-ga7de91bf-1jammy_amd64.deb ... 2023-10-03T17:33:07.533 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking ceph-osd (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-03T17:33:09.637 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package ceph. 2023-10-03T17:33:09.652 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../059-ceph_18.0.0-6336-ga7de91bf-1jammy_amd64.deb ... 2023-10-03T17:33:09.695 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking ceph (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-03T17:33:09.956 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package ceph-fuse. 2023-10-03T17:33:09.971 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../060-ceph-fuse_18.0.0-6336-ga7de91bf-1jammy_amd64.deb ... 2023-10-03T17:33:10.006 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking ceph-fuse (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-03T17:33:10.350 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package ceph-mds. 2023-10-03T17:33:10.377 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../061-ceph-mds_18.0.0-6336-ga7de91bf-1jammy_amd64.deb ... 2023-10-03T17:33:10.417 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking ceph-mds (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-03T17:33:10.804 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package cephadm. 2023-10-03T17:33:10.830 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../062-cephadm_18.0.0-6336-ga7de91bf-1jammy_amd64.deb ... 2023-10-03T17:33:10.870 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking cephadm (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-03T17:33:11.139 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-asyncssh. 2023-10-03T17:33:11.159 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../063-python3-asyncssh_2.5.0-1_all.deb ... 2023-10-03T17:33:11.189 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2023-10-03T17:33:11.500 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python-babel-localedata. 2023-10-03T17:33:11.527 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../064-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2023-10-03T17:33:11.570 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2023-10-03T17:33:12.547 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-babel. 2023-10-03T17:33:12.574 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../065-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2023-10-03T17:33:12.613 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2023-10-03T17:33:12.975 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-jinja2. 2023-10-03T17:33:13.002 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../066-python3-jinja2_3.0.3-1_all.deb ... 2023-10-03T17:33:13.041 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-jinja2 (3.0.3-1) ... 2023-10-03T17:33:13.395 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2023-10-03T17:33:13.422 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../067-ceph-mgr-cephadm_18.0.0-6336-ga7de91bf-1jammy_all.deb ... 2023-10-03T17:33:13.461 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking ceph-mgr-cephadm (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-03T17:33:13.798 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-bcrypt. 2023-10-03T17:33:13.825 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../068-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2023-10-03T17:33:13.865 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2023-10-03T17:33:14.168 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-repoze.lru. 2023-10-03T17:33:14.194 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../069-python3-repoze.lru_0.7-2_all.deb ... 2023-10-03T17:33:14.234 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2023-10-03T17:33:14.479 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-routes. 2023-10-03T17:33:14.494 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../070-python3-routes_2.5.1-1ubuntu1_all.deb ... 2023-10-03T17:33:14.537 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2023-10-03T17:33:14.832 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2023-10-03T17:33:14.859 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../071-ceph-mgr-dashboard_18.0.0-6336-ga7de91bf-1jammy_all.deb ... 2023-10-03T17:33:14.890 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking ceph-mgr-dashboard (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-03T17:33:15.327 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2023-10-03T17:33:15.342 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../072-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2023-10-03T17:33:15.385 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2023-10-03T17:33:15.872 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-joblib. 2023-10-03T17:33:15.888 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../073-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2023-10-03T17:33:15.922 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2023-10-03T17:33:16.284 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-threadpoolctl. 2023-10-03T17:33:16.310 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../074-python3-threadpoolctl_3.1.0-1_all.deb ... 2023-10-03T17:33:16.350 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2023-10-03T17:33:16.653 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-sklearn. 2023-10-03T17:33:16.682 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../075-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2023-10-03T17:33:16.721 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2023-10-03T17:33:17.405 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2023-10-03T17:33:17.433 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../076-ceph-mgr-diskprediction-local_18.0.0-6336-ga7de91bf-1jammy_all.deb ... 2023-10-03T17:33:17.480 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking ceph-mgr-diskprediction-local (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-03T17:33:18.433 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-cachetools. 2023-10-03T17:33:18.460 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../077-python3-cachetools_5.0.0-1_all.deb ... 2023-10-03T17:33:18.499 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2023-10-03T17:33:18.802 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-pyasn1. 2023-10-03T17:33:18.829 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../078-python3-pyasn1_0.4.8-1_all.deb ... 2023-10-03T17:33:18.869 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2023-10-03T17:33:19.214 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-pyasn1-modules. 2023-10-03T17:33:19.241 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../079-python3-pyasn1-modules_0.2.1-1_all.deb ... 2023-10-03T17:33:19.281 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2023-10-03T17:33:19.592 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-rsa. 2023-10-03T17:33:19.619 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../080-python3-rsa_4.8-1_all.deb ... 2023-10-03T17:33:19.659 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-rsa (4.8-1) ... 2023-10-03T17:33:19.937 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-google-auth. 2023-10-03T17:33:19.964 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../081-python3-google-auth_1.5.1-3_all.deb ... 2023-10-03T17:33:19.995 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2023-10-03T17:33:20.265 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-requests-oauthlib. 2023-10-03T17:33:20.281 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../082-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2023-10-03T17:33:20.323 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2023-10-03T17:33:20.736 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-websocket. 2023-10-03T17:33:20.764 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../083-python3-websocket_1.2.3-1_all.deb ... 2023-10-03T17:33:20.803 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-websocket (1.2.3-1) ... 2023-10-03T17:33:21.123 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-kubernetes. 2023-10-03T17:33:21.150 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../084-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2023-10-03T17:33:21.190 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2023-10-03T17:33:21.779 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2023-10-03T17:33:21.807 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../085-ceph-mgr-k8sevents_18.0.0-6336-ga7de91bf-1jammy_all.deb ... 2023-10-03T17:33:21.845 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking ceph-mgr-k8sevents (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-03T17:33:22.190 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package libonig5:amd64. 2023-10-03T17:33:22.217 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../086-libonig5_6.9.7.1-2build1_amd64.deb ... 2023-10-03T17:33:22.257 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2023-10-03T17:33:22.601 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package libjq1:amd64. 2023-10-03T17:33:22.628 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../087-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2023-10-03T17:33:22.668 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2023-10-03T17:33:22.895 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package jq. 2023-10-03T17:33:22.911 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../088-jq_1.6-2.1ubuntu3_amd64.deb ... 2023-10-03T17:33:22.953 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2023-10-03T17:33:23.206 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package socat. 2023-10-03T17:33:23.222 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../089-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2023-10-03T17:33:23.265 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2023-10-03T17:33:23.584 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package xmlstarlet. 2023-10-03T17:33:23.600 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../090-xmlstarlet_1.6.1-2.1_amd64.deb ... 2023-10-03T17:33:23.643 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2023-10-03T17:33:23.997 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package ceph-test. 2023-10-03T17:33:24.013 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../091-ceph-test_18.0.0-6336-ga7de91bf-1jammy_amd64.deb ... 2023-10-03T17:33:24.064 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking ceph-test (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-03T17:33:26.010 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package ceph-volume. 2023-10-03T17:33:26.037 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../092-ceph-volume_18.0.0-6336-ga7de91bf-1jammy_all.deb ... 2023-10-03T17:33:26.076 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking ceph-volume (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-03T17:33:26.422 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package libcephfs-dev. 2023-10-03T17:33:26.450 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../093-libcephfs-dev_18.0.0-6336-ga7de91bf-1jammy_amd64.deb ... 2023-10-03T17:33:26.489 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking libcephfs-dev (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-03T17:33:26.792 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package nvme-cli. 2023-10-03T17:33:26.820 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../094-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2023-10-03T17:33:26.858 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2023-10-03T17:33:27.181 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python-asyncssh-doc. 2023-10-03T17:33:27.209 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../095-python-asyncssh-doc_2.5.0-1_all.deb ... 2023-10-03T17:33:27.247 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2023-10-03T17:33:27.651 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-iniconfig. 2023-10-03T17:33:27.678 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../096-python3-iniconfig_1.1.1-2_all.deb ... 2023-10-03T17:33:27.717 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2023-10-03T17:33:28.020 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-pastescript. 2023-10-03T17:33:28.047 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../097-python3-pastescript_2.0.2-4_all.deb ... 2023-10-03T17:33:28.086 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2023-10-03T17:33:28.415 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-pluggy. 2023-10-03T17:33:28.443 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../098-python3-pluggy_0.13.0-7.1_all.deb ... 2023-10-03T17:33:28.473 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2023-10-03T17:33:28.882 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-psutil. 2023-10-03T17:33:28.910 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../099-python3-psutil_5.9.0-1build1_amd64.deb ... 2023-10-03T17:33:29.101 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2023-10-03T17:33:29.429 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-py. 2023-10-03T17:33:29.457 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../100-python3-py_1.10.0-1_all.deb ... 2023-10-03T17:33:29.495 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-py (1.10.0-1) ... 2023-10-03T17:33:29.816 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-pygments. 2023-10-03T17:33:29.844 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../101-python3-pygments_2.11.2+dfsg-2_all.deb ... 2023-10-03T17:33:29.882 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2023-10-03T17:33:30.370 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-pyinotify. 2023-10-03T17:33:30.398 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../102-python3-pyinotify_0.9.6-1.3_all.deb ... 2023-10-03T17:33:30.436 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2023-10-03T17:33:30.697 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-toml. 2023-10-03T17:33:30.725 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../103-python3-toml_0.10.2-1_all.deb ... 2023-10-03T17:33:30.755 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-toml (0.10.2-1) ... 2023-10-03T17:33:31.058 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-pytest. 2023-10-03T17:33:31.086 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../104-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2023-10-03T17:33:31.125 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2023-10-03T17:33:31.420 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-simplejson. 2023-10-03T17:33:31.448 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../105-python3-simplejson_3.17.6-1build1_amd64.deb ... 2023-10-03T17:33:31.478 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2023-10-03T17:33:31.723 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package qttranslations5-l10n. 2023-10-03T17:33:31.750 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../106-qttranslations5-l10n_5.15.3-1_all.deb ... 2023-10-03T17:33:31.798 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2023-10-03T17:33:32.527 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package radosgw. 2023-10-03T17:33:32.554 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../107-radosgw_18.0.0-6336-ga7de91bf-1jammy_amd64.deb ... 2023-10-03T17:33:32.593 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking radosgw (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-03T17:33:33.363 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package rbd-fuse. 2023-10-03T17:33:33.390 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../108-rbd-fuse_18.0.0-6336-ga7de91bf-1jammy_amd64.deb ... 2023-10-03T17:33:33.429 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking rbd-fuse (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-03T17:33:33.713 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2023-10-03T17:33:33.984 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2023-10-03T17:33:34.110 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2023-10-03T17:33:34.368 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2023-10-03T17:33:34.711 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2023-10-03T17:33:35.198 INFO:teuthology.orchestra.run.smithi111.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2023-10-03T17:33:35.279 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up cephadm (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-03T17:33:35.730 INFO:teuthology.orchestra.run.smithi111.stdout:Adding system user cephadm....done 2023-10-03T17:33:35.885 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2023-10-03T17:33:36.220 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2023-10-03T17:33:36.463 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2023-10-03T17:33:36.564 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2023-10-03T17:33:36.807 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-repoze.lru (0.7-2) ... 2023-10-03T17:33:37.058 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2023-10-03T17:33:37.184 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-py (1.10.0-1) ... 2023-10-03T17:33:37.486 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2023-10-03T17:33:37.846 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-cachetools (5.0.0-1) ... 2023-10-03T17:33:38.089 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-openssl (21.0.0-1) ... 2023-10-03T17:33:38.357 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2023-10-03T17:33:38.616 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2023-10-03T17:33:38.876 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2023-10-03T17:33:39.135 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-ceph-argparse (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-03T17:33:39.394 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2023-10-03T17:33:39.520 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2023-10-03T17:33:39.803 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2023-10-03T17:33:39.906 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2023-10-03T17:33:40.266 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2023-10-03T17:33:40.392 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-natsort (8.0.2-1) ... 2023-10-03T17:33:40.659 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2023-10-03T17:33:40.927 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2023-10-03T17:33:41.236 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2023-10-03T17:33:41.905 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2023-10-03T17:33:42.165 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2023-10-03T17:33:42.282 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2023-10-03T17:33:42.408 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2023-10-03T17:33:42.710 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2023-10-03T17:33:43.094 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-certifi (2020.6.20-1) ... 2023-10-03T17:33:43.345 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2023-10-03T17:33:43.713 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2023-10-03T17:33:43.981 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2023-10-03T17:33:44.366 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2023-10-03T17:33:44.643 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2023-10-03T17:33:44.769 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-ceph-common (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-03T17:33:45.070 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-idna (3.3-1) ... 2023-10-03T17:33:45.371 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2023-10-03T17:33:46.502 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-03T17:33:46.636 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-toml (0.10.2-1) ... 2023-10-03T17:33:46.887 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2023-10-03T17:33:46.997 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2023-10-03T17:33:47.098 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-urllib3 (1.26.5-1~exp1) ... 2023-10-03T17:33:47.407 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2023-10-03T17:33:47.692 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2023-10-03T17:33:47.943 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-03T17:33:48.069 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up libfmt8:amd64 (8.1.1+ds1-2) ... 2023-10-03T17:33:48.178 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2023-10-03T17:33:48.463 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2023-10-03T17:33:48.714 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-logutils (0.3.3-8) ... 2023-10-03T17:33:48.982 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-tempora (4.1.2-1) ... 2023-10-03T17:33:49.242 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2023-10-03T17:33:49.529 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-prettytable (2.5.0-2) ... 2023-10-03T17:33:50.043 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2023-10-03T17:33:50.161 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-websocket (1.2.3-1) ... 2023-10-03T17:33:50.455 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2023-10-03T17:33:50.555 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2023-10-03T17:33:50.832 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2023-10-03T17:33:51.133 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2023-10-03T17:33:51.384 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2023-10-03T17:33:51.785 INFO:teuthology.orchestra.run.smithi111.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2023-10-03T17:33:52.029 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2023-10-03T17:33:52.154 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2023-10-03T17:33:52.539 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2023-10-03T17:33:52.799 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-portend (3.0.0-1) ... 2023-10-03T17:33:53.058 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-03T17:33:53.184 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-jinja2 (3.0.3-1) ... 2023-10-03T17:33:53.510 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2023-10-03T17:33:53.803 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2023-10-03T17:33:54.054 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2023-10-03T17:33:54.172 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-rsa (4.8-1) ... 2023-10-03T17:33:54.448 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-webtest (2.0.35-1) ... 2023-10-03T17:33:54.783 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2023-10-03T17:33:55.050 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2023-10-03T17:33:55.469 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-pastescript (2.0.2-4) ... 2023-10-03T17:33:55.770 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2023-10-03T17:33:56.114 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2023-10-03T17:33:56.240 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up librados2 (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-03T17:33:56.366 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up librgw2 (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-03T17:33:56.492 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up libsqlite3-mod-ceph (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-03T17:33:56.618 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up libcephfs2 (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-03T17:33:56.744 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up libradosstriper1 (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-03T17:33:56.947 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-google-auth (1.5.1-3) ... 2023-10-03T17:33:57.225 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up librbd1 (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-03T17:33:57.350 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up ceph-mgr-modules-core (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-03T17:33:57.476 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up ceph-fuse (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-03T17:33:57.706 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-10-03T17:33:57.706 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-10-03T17:33:58.238 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up libcephfs-dev (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-03T17:33:58.345 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-rados (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-03T17:33:58.447 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2023-10-03T17:33:59.642 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-rbd (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-03T17:33:59.769 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up rbd-fuse (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-03T17:33:59.894 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-rgw (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-03T17:34:00.163 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-cephfs (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-03T17:34:00.355 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up ceph-common (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-03T17:34:00.839 INFO:teuthology.orchestra.run.smithi111.stdout:Adding group ceph....done 2023-10-03T17:34:01.199 INFO:teuthology.orchestra.run.smithi111.stdout:Adding system user ceph....done 2023-10-03T17:34:01.317 INFO:teuthology.orchestra.run.smithi111.stdout:Setting system user ceph properties....done 2023-10-03T17:34:01.330 INFO:teuthology.orchestra.run.smithi111.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2023-10-03T17:34:01.487 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2023-10-03T17:34:01.870 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2023-10-03T17:34:02.414 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up ceph-test (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-03T17:34:02.558 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up radosgw (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-03T17:34:02.833 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-10-03T17:34:02.833 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-10-03T17:34:03.677 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up ceph-base (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-03T17:34:04.017 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2023-10-03T17:34:04.528 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up ceph-mds (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-03T17:34:04.737 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-10-03T17:34:04.738 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-10-03T17:34:05.299 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up ceph-mgr (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-03T17:34:05.617 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-10-03T17:34:05.618 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-10-03T17:34:06.113 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up ceph-osd (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-03T17:34:06.426 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-10-03T17:34:06.426 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-10-03T17:34:06.988 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up ceph-mgr-k8sevents (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-03T17:34:07.106 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up ceph-mgr-diskprediction-local (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-03T17:34:07.257 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up ceph-mon (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-03T17:34:07.478 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-10-03T17:34:07.478 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-10-03T17:34:07.967 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up ceph-mgr-cephadm (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-03T17:34:08.118 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up ceph (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-03T17:34:08.236 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up ceph-mgr-dashboard (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-03T17:34:08.363 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up ceph-volume (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-03T17:34:08.464 INFO:teuthology.orchestra.run.smithi111.stdout:Processing triggers for man-db (2.10.2-1) ... 2023-10-03T17:34:09.456 INFO:teuthology.orchestra.run.smithi111.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2023-10-03T17:34:09.922 INFO:teuthology.orchestra.run.smithi111.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-10-03T17:34:09.925 DEBUG:teuthology.parallel:result is None 2023-10-03T17:34:09.925 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=a7de91bfa5778a0d559a7e9bc8e35f93456b41ad 2023-10-03T17:34:10.151 DEBUG:teuthology.orchestra.run.smithi111:> dpkg-query -W -f '${Version}' ceph 2023-10-03T17:34:10.170 INFO:teuthology.orchestra.run.smithi111.stdout:18.0.0-6336-ga7de91bf-1jammy 2023-10-03T17:34:10.170 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-6336-ga7de91bf-1jammy 2023-10-03T17:34:10.170 INFO:teuthology.task.install:The correct ceph version 18.0.0-6336-ga7de91bf-1jammy is installed. 2023-10-03T17:34:10.171 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-10-03T17:34:10.171 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-10-03T17:34:10.171 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-10-03T17:34:10.226 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-10-03T17:34:10.226 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-10-03T17:34:10.226 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd of=/usr/bin/daemon-helper 2023-10-03T17:34:10.278 DEBUG:teuthology.orchestra.run.smithi111:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-10-03T17:34:10.334 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-10-03T17:34:10.334 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-10-03T17:34:10.335 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd of=/usr/bin/adjust-ulimits 2023-10-03T17:34:10.390 DEBUG:teuthology.orchestra.run.smithi111:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-10-03T17:34:10.446 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-10-03T17:34:10.446 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-10-03T17:34:10.446 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd of=/usr/bin/stdin-killer 2023-10-03T17:34:10.499 DEBUG:teuthology.orchestra.run.smithi111:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-10-03T17:34:10.554 INFO:teuthology.run_tasks:Running task ceph... 2023-10-03T17:34:10.629 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-10-03T17:34:10.630 DEBUG:teuthology.orchestra.run.smithi111:> sudo chmod 777 /var/log/ceph 2023-10-03T17:34:10.642 INFO:tasks.ceph:Disabling ceph logrotate... 2023-10-03T17:34:10.642 DEBUG:teuthology.orchestra.run.smithi111:> sudo rm -f -- /etc/logrotate.d/ceph 2023-10-03T17:34:10.698 INFO:tasks.ceph:Creating extra log directories... 2023-10-03T17:34:10.698 DEBUG:teuthology.orchestra.run.smithi111:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-10-03T17:34:10.762 INFO:tasks.ceph:Creating ceph cluster ceph... 2023-10-03T17:34:10.762 INFO:tasks.ceph:config {'conf': {'client': {'rbd default data pool': 'datapool', 'rbd default features': 1}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1, 'debug rbd': 20}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zlib', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-10-03T17:34:10.762 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/rraja-2023-10-03_17:10:36-rbd:cli-wip-ajarr-63009-only-reacquire-kickstart-distro-default-smithi/7409699', 'branch': 'wip-ajarr-63009-only-reacquire-kickstart', 'description': 'rbd:cli/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} features/layering msgr-failures/few objectstore/bluestore-comp-zlib pool/replicated-data-pool supported-random-distro$/{ubuntu_latest} workloads/rbd_support_module_recovery}', 'email': 'rraja@redhat.com', 'first_in_suite': False, 'job_id': '7409699', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'rraja-2023-10-03_17:10:36-rbd:cli-wip-ajarr-63009-only-reacquire-kickstart-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 3, 'size': 30}}], 'os_type': 'ubuntu', 'os_version': '22.04', 'overrides': {'admin_socket': {'branch': 'wip-ajarr-63009-only-reacquire-kickstart'}, 'ceph': {'conf': {'client': {'rbd default data pool': 'datapool', 'rbd default features': 1}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1, 'debug rbd': 20}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zlib', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': 'a7de91bfa5778a0d559a7e9bc8e35f93456b41ad'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'global': {'osd crush chooseleaf type': 0, 'osd pool default pg num': 128, 'osd pool default pgp num': 128, 'osd pool default size': 2}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'a7de91bfa5778a0d559a7e9bc8e35f93456b41ad'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-ajarr-63009-only-reacquire-kickstart', 'sha1': 'a7de91bfa5778a0d559a7e9bc8e35f93456b41ad'}}, 'owner': 'scheduled_rraja@teuthology', 'priority': 75, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0']], 'seed': 8213, 'sha1': 'a7de91bfa5778a0d559a7e9bc8e35f93456b41ad', 'sleep_before_teardown': 0, 'suite': 'rbd:cli', 'suite_branch': 'wip-ajarr-63009-only-reacquire-kickstart', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_a7de91bfa5778a0d559a7e9bc8e35f93456b41ad/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': 'a7de91bfa5778a0d559a7e9bc8e35f93456b41ad', 'targets': {'smithi111.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEnvb/71JGXyR+c0QUpVhUkGSLg+nELPCl6VdIC0fmBzqofvlUwDqAZ884uaUDsJS88B0iMGAbcU5dgT0Qs4M/A='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'exec': {'client.0': ['sudo ceph osd pool create datapool 4', 'rbd pool init datapool']}}, {'install': {'extra_system_packages': ['fio']}}, {'workunit': {'clients': {'client.0': ['rbd/rbd_support_module_recovery.sh']}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '54e62bcbac4e53d9685e08328b790d3b20d71cae', 'timestamp': '2023-10-03_17:10:36', 'tube': 'smithi', 'user': 'rraja', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2830'} 2023-10-03T17:34:10.763 DEBUG:teuthology.orchestra.run.smithi111:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-10-03T17:34:10.804 DEBUG:teuthology.orchestra.run.smithi111:> sudo install -d -m0777 -- /var/run/ceph 2023-10-03T17:34:10.863 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-10-03T17:34:10.864 DEBUG:teuthology.orchestra.run.smithi111:> dd if=/scratch_devs of=/dev/stdout 2023-10-03T17:34:10.911 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-10-03T17:34:10.911 DEBUG:teuthology.orchestra.run.smithi111:> stat /dev/vg_nvme/lv_1 2023-10-03T17:34:10.959 INFO:teuthology.orchestra.run.smithi111.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-10-03T17:34:10.959 INFO:teuthology.orchestra.run.smithi111.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-03T17:34:10.959 INFO:teuthology.orchestra.run.smithi111.stdout:Device: 5h/5d Inode: 882 Links: 1 2023-10-03T17:34:10.960 INFO:teuthology.orchestra.run.smithi111.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-03T17:34:10.960 INFO:teuthology.orchestra.run.smithi111.stdout:Access: 2023-10-03 17:28:43.828871193 +0000 2023-10-03T17:34:10.960 INFO:teuthology.orchestra.run.smithi111.stdout:Modify: 2023-10-03 17:28:43.028878339 +0000 2023-10-03T17:34:10.960 INFO:teuthology.orchestra.run.smithi111.stdout:Change: 2023-10-03 17:28:43.028878339 +0000 2023-10-03T17:34:10.960 INFO:teuthology.orchestra.run.smithi111.stdout: Birth: - 2023-10-03T17:34:10.960 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-10-03T17:34:11.013 INFO:teuthology.orchestra.run.smithi111.stderr:1+0 records in 2023-10-03T17:34:11.013 INFO:teuthology.orchestra.run.smithi111.stderr:1+0 records out 2023-10-03T17:34:11.013 INFO:teuthology.orchestra.run.smithi111.stderr:512 bytes copied, 0.000397008 s, 1.3 MB/s 2023-10-03T17:34:11.014 DEBUG:teuthology.orchestra.run.smithi111:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-10-03T17:34:11.065 DEBUG:teuthology.orchestra.run.smithi111:> stat /dev/vg_nvme/lv_2 2023-10-03T17:34:11.111 INFO:teuthology.orchestra.run.smithi111.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-10-03T17:34:11.112 INFO:teuthology.orchestra.run.smithi111.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-03T17:34:11.112 INFO:teuthology.orchestra.run.smithi111.stdout:Device: 5h/5d Inode: 895 Links: 1 2023-10-03T17:34:11.112 INFO:teuthology.orchestra.run.smithi111.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-03T17:34:11.112 INFO:teuthology.orchestra.run.smithi111.stdout:Access: 2023-10-03 17:28:44.880861796 +0000 2023-10-03T17:34:11.112 INFO:teuthology.orchestra.run.smithi111.stdout:Modify: 2023-10-03 17:28:44.052869192 +0000 2023-10-03T17:34:11.112 INFO:teuthology.orchestra.run.smithi111.stdout:Change: 2023-10-03 17:28:44.052869192 +0000 2023-10-03T17:34:11.112 INFO:teuthology.orchestra.run.smithi111.stdout: Birth: - 2023-10-03T17:34:11.112 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-10-03T17:34:11.166 INFO:teuthology.orchestra.run.smithi111.stderr:1+0 records in 2023-10-03T17:34:11.166 INFO:teuthology.orchestra.run.smithi111.stderr:1+0 records out 2023-10-03T17:34:11.166 INFO:teuthology.orchestra.run.smithi111.stderr:512 bytes copied, 0.000424131 s, 1.2 MB/s 2023-10-03T17:34:11.167 DEBUG:teuthology.orchestra.run.smithi111:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-10-03T17:34:11.216 DEBUG:teuthology.orchestra.run.smithi111:> stat /dev/vg_nvme/lv_3 2023-10-03T17:34:11.263 INFO:teuthology.orchestra.run.smithi111.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-10-03T17:34:11.263 INFO:teuthology.orchestra.run.smithi111.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-03T17:34:11.263 INFO:teuthology.orchestra.run.smithi111.stdout:Device: 5h/5d Inode: 916 Links: 1 2023-10-03T17:34:11.263 INFO:teuthology.orchestra.run.smithi111.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-03T17:34:11.263 INFO:teuthology.orchestra.run.smithi111.stdout:Access: 2023-10-03 17:28:45.864853007 +0000 2023-10-03T17:34:11.263 INFO:teuthology.orchestra.run.smithi111.stdout:Modify: 2023-10-03 17:28:45.132859545 +0000 2023-10-03T17:34:11.264 INFO:teuthology.orchestra.run.smithi111.stdout:Change: 2023-10-03 17:28:45.132859545 +0000 2023-10-03T17:34:11.264 INFO:teuthology.orchestra.run.smithi111.stdout: Birth: - 2023-10-03T17:34:11.264 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-10-03T17:34:11.318 INFO:teuthology.orchestra.run.smithi111.stderr:1+0 records in 2023-10-03T17:34:11.318 INFO:teuthology.orchestra.run.smithi111.stderr:1+0 records out 2023-10-03T17:34:11.318 INFO:teuthology.orchestra.run.smithi111.stderr:512 bytes copied, 0.000380053 s, 1.3 MB/s 2023-10-03T17:34:11.319 DEBUG:teuthology.orchestra.run.smithi111:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-10-03T17:34:11.372 DEBUG:teuthology.orchestra.run.smithi111:> stat /dev/vg_nvme/lv_4 2023-10-03T17:34:11.423 INFO:teuthology.orchestra.run.smithi111.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-10-03T17:34:11.423 INFO:teuthology.orchestra.run.smithi111.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-03T17:34:11.423 INFO:teuthology.orchestra.run.smithi111.stdout:Device: 5h/5d Inode: 930 Links: 1 2023-10-03T17:34:11.423 INFO:teuthology.orchestra.run.smithi111.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-03T17:34:11.423 INFO:teuthology.orchestra.run.smithi111.stdout:Access: 2023-10-03 17:28:46.772844895 +0000 2023-10-03T17:34:11.423 INFO:teuthology.orchestra.run.smithi111.stdout:Modify: 2023-10-03 17:28:46.104850863 +0000 2023-10-03T17:34:11.423 INFO:teuthology.orchestra.run.smithi111.stdout:Change: 2023-10-03 17:28:46.104850863 +0000 2023-10-03T17:34:11.423 INFO:teuthology.orchestra.run.smithi111.stdout: Birth: - 2023-10-03T17:34:11.424 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-10-03T17:34:11.477 INFO:teuthology.orchestra.run.smithi111.stderr:1+0 records in 2023-10-03T17:34:11.477 INFO:teuthology.orchestra.run.smithi111.stderr:1+0 records out 2023-10-03T17:34:11.477 INFO:teuthology.orchestra.run.smithi111.stderr:512 bytes copied, 0.000415036 s, 1.2 MB/s 2023-10-03T17:34:11.478 DEBUG:teuthology.orchestra.run.smithi111:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-10-03T17:34:11.533 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-10-03T17:34:11.533 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi111.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2023-10-03T17:34:11.533 INFO:tasks.ceph:Generating config... 2023-10-03T17:34:11.534 INFO:tasks.ceph:[client] rbd default data pool = datapool 2023-10-03T17:34:11.534 INFO:tasks.ceph:[client] rbd default features = 1 2023-10-03T17:34:11.534 INFO:tasks.ceph:[global] mon client directed command retry = 5 2023-10-03T17:34:11.534 INFO:tasks.ceph:[global] mon warn on pool no app = False 2023-10-03T17:34:11.534 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2023-10-03T17:34:11.534 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-10-03T17:34:11.534 INFO:tasks.ceph:[mgr] debug ms = 1 2023-10-03T17:34:11.535 INFO:tasks.ceph:[mgr] debug rbd = 20 2023-10-03T17:34:11.535 INFO:tasks.ceph:[mon] debug mon = 20 2023-10-03T17:34:11.535 INFO:tasks.ceph:[mon] debug ms = 1 2023-10-03T17:34:11.535 INFO:tasks.ceph:[mon] debug paxos = 20 2023-10-03T17:34:11.535 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-10-03T17:34:11.535 INFO:tasks.ceph:[osd] bluestore compression algorithm = zlib 2023-10-03T17:34:11.535 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2023-10-03T17:34:11.535 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-10-03T17:34:11.535 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2023-10-03T17:34:11.535 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2023-10-03T17:34:11.535 INFO:tasks.ceph:[osd] debug ms = 1 2023-10-03T17:34:11.535 INFO:tasks.ceph:[osd] debug osd = 20 2023-10-03T17:34:11.536 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2023-10-03T17:34:11.536 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-10-03T17:34:11.536 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-10-03T17:34:11.536 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-10-03T17:34:11.536 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-10-03T17:34:11.536 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-10-03T17:34:11.536 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2023-10-03T17:34:11.536 INFO:tasks.ceph:Setting up mon.a... 2023-10-03T17:34:11.536 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2023-10-03T17:34:11.605 INFO:teuthology.orchestra.run.smithi111.stdout:creating /etc/ceph/ceph.keyring 2023-10-03T17:34:11.615 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2023-10-03T17:34:11.691 DEBUG:teuthology.orchestra.run.smithi111:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-10-03T17:34:11.750 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.111')] 2023-10-03T17:34:11.750 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.111', 'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zlib', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug rbd': 20}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'rbd default data pool': 'datapool', 'rbd default features': 1}, 'mon.a': {}} 2023-10-03T17:34:11.751 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-10-03T17:34:11.751 DEBUG:teuthology.orchestra.run.smithi111:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-10-03T17:34:11.799 DEBUG:teuthology.orchestra.run.smithi111:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.111 --print /home/ubuntu/cephtest/ceph.monmap 2023-10-03T17:34:11.880 INFO:teuthology.orchestra.run.smithi111.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2023-10-03T17:34:11.880 INFO:teuthology.orchestra.run.smithi111.stdout:monmaptool: generated fsid 499d6abd-e417-4d8d-a4aa-a7db52cf5eb1 2023-10-03T17:34:11.880 INFO:teuthology.orchestra.run.smithi111.stdout:setting min_mon_release = pacific 2023-10-03T17:34:11.880 INFO:teuthology.orchestra.run.smithi111.stdout:epoch 0 2023-10-03T17:34:11.881 INFO:teuthology.orchestra.run.smithi111.stdout:fsid 499d6abd-e417-4d8d-a4aa-a7db52cf5eb1 2023-10-03T17:34:11.881 INFO:teuthology.orchestra.run.smithi111.stdout:last_changed 2023-10-03T17:34:11.882835+0000 2023-10-03T17:34:11.881 INFO:teuthology.orchestra.run.smithi111.stdout:created 2023-10-03T17:34:11.882835+0000 2023-10-03T17:34:11.881 INFO:teuthology.orchestra.run.smithi111.stdout:min_mon_release 16 (pacific) 2023-10-03T17:34:11.881 INFO:teuthology.orchestra.run.smithi111.stdout:election_strategy: 1 2023-10-03T17:34:11.881 INFO:teuthology.orchestra.run.smithi111.stdout:0: [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] mon.a 2023-10-03T17:34:11.881 INFO:teuthology.orchestra.run.smithi111.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2023-10-03T17:34:11.882 DEBUG:teuthology.orchestra.run.smithi111:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-10-03T17:34:11.927 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 499d6abd-e417-4d8d-a4aa-a7db52cf5eb1... 2023-10-03T17:34:11.928 DEBUG:teuthology.orchestra.run.smithi111:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-10-03T17:34:12.000 INFO:teuthology.orchestra.run.smithi111.stdout:[global] 2023-10-03T17:34:12.000 INFO:teuthology.orchestra.run.smithi111.stdout: chdir = "" 2023-10-03T17:34:12.001 INFO:teuthology.orchestra.run.smithi111.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-10-03T17:34:12.001 INFO:teuthology.orchestra.run.smithi111.stdout: auth supported = cephx 2023-10-03T17:34:12.001 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-03T17:34:12.001 INFO:teuthology.orchestra.run.smithi111.stdout: filestore xattr use omap = true 2023-10-03T17:34:12.001 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-03T17:34:12.001 INFO:teuthology.orchestra.run.smithi111.stdout: mon clock drift allowed = 1.000 2023-10-03T17:34:12.001 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-03T17:34:12.001 INFO:teuthology.orchestra.run.smithi111.stdout: osd crush chooseleaf type = 0 2023-10-03T17:34:12.001 INFO:teuthology.orchestra.run.smithi111.stdout: auth debug = true 2023-10-03T17:34:12.001 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-03T17:34:12.002 INFO:teuthology.orchestra.run.smithi111.stdout: ms die on old message = true 2023-10-03T17:34:12.002 INFO:teuthology.orchestra.run.smithi111.stdout: ms die on bug = true 2023-10-03T17:34:12.002 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-03T17:34:12.002 INFO:teuthology.orchestra.run.smithi111.stdout: mon max pg per osd = 10000 # >= luminous 2023-10-03T17:34:12.002 INFO:teuthology.orchestra.run.smithi111.stdout: mon pg warn max object skew = 0 2023-10-03T17:34:12.002 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-03T17:34:12.002 INFO:teuthology.orchestra.run.smithi111.stdout: # disable pg_autoscaler by default for new pools 2023-10-03T17:34:12.002 INFO:teuthology.orchestra.run.smithi111.stdout: osd_pool_default_pg_autoscale_mode = off 2023-10-03T17:34:12.002 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-03T17:34:12.002 INFO:teuthology.orchestra.run.smithi111.stdout: osd pool default size = 2 2023-10-03T17:34:12.002 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-03T17:34:12.002 INFO:teuthology.orchestra.run.smithi111.stdout: mon osd allow primary affinity = true 2023-10-03T17:34:12.003 INFO:teuthology.orchestra.run.smithi111.stdout: mon osd allow pg remap = true 2023-10-03T17:34:12.003 INFO:teuthology.orchestra.run.smithi111.stdout: mon warn on legacy crush tunables = false 2023-10-03T17:34:12.003 INFO:teuthology.orchestra.run.smithi111.stdout: mon warn on crush straw calc version zero = false 2023-10-03T17:34:12.003 INFO:teuthology.orchestra.run.smithi111.stdout: mon warn on no sortbitwise = false 2023-10-03T17:34:12.003 INFO:teuthology.orchestra.run.smithi111.stdout: mon warn on osd down out interval zero = false 2023-10-03T17:34:12.003 INFO:teuthology.orchestra.run.smithi111.stdout: mon warn on too few osds = false 2023-10-03T17:34:12.003 INFO:teuthology.orchestra.run.smithi111.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-10-03T17:34:12.003 INFO:teuthology.orchestra.run.smithi111.stdout: mon_warn_on_pool_no_redundancy = false 2023-10-03T17:34:12.003 INFO:teuthology.orchestra.run.smithi111.stdout: mon_allow_pool_size_one = true 2023-10-03T17:34:12.003 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-03T17:34:12.004 INFO:teuthology.orchestra.run.smithi111.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-10-03T17:34:12.004 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-03T17:34:12.004 INFO:teuthology.orchestra.run.smithi111.stdout: osd default data pool replay window = 5 2023-10-03T17:34:12.004 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-03T17:34:12.004 INFO:teuthology.orchestra.run.smithi111.stdout: mon allow pool delete = true 2023-10-03T17:34:12.004 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-03T17:34:12.004 INFO:teuthology.orchestra.run.smithi111.stdout: mon cluster log file level = debug 2023-10-03T17:34:12.004 INFO:teuthology.orchestra.run.smithi111.stdout: debug asserts on shutdown = true 2023-10-03T17:34:12.004 INFO:teuthology.orchestra.run.smithi111.stdout: mon health detail to clog = false 2023-10-03T17:34:12.004 INFO:teuthology.orchestra.run.smithi111.stdout: mon host = 172.21.15.111 2023-10-03T17:34:12.005 INFO:teuthology.orchestra.run.smithi111.stdout: mon client directed command retry = 5 2023-10-03T17:34:12.005 INFO:teuthology.orchestra.run.smithi111.stdout: mon warn on pool no app = False 2023-10-03T17:34:12.005 INFO:teuthology.orchestra.run.smithi111.stdout: ms inject socket failures = 5000 2023-10-03T17:34:12.005 INFO:teuthology.orchestra.run.smithi111.stdout: fsid = 499d6abd-e417-4d8d-a4aa-a7db52cf5eb1 2023-10-03T17:34:12.005 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-03T17:34:12.005 INFO:teuthology.orchestra.run.smithi111.stdout:[osd] 2023-10-03T17:34:12.005 INFO:teuthology.orchestra.run.smithi111.stdout: osd journal size = 100 2023-10-03T17:34:12.005 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-03T17:34:12.005 INFO:teuthology.orchestra.run.smithi111.stdout: osd scrub load threshold = 5.0 2023-10-03T17:34:12.005 INFO:teuthology.orchestra.run.smithi111.stdout: osd scrub max interval = 600 2023-10-03T17:34:12.006 INFO:teuthology.orchestra.run.smithi111.stdout: osd mclock profile = high_recovery_ops 2023-10-03T17:34:12.006 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-03T17:34:12.006 INFO:teuthology.orchestra.run.smithi111.stdout: osd recover clone overlap = true 2023-10-03T17:34:12.006 INFO:teuthology.orchestra.run.smithi111.stdout: osd recovery max chunk = 1048576 2023-10-03T17:34:12.006 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-03T17:34:12.006 INFO:teuthology.orchestra.run.smithi111.stdout: osd debug shutdown = true 2023-10-03T17:34:12.006 INFO:teuthology.orchestra.run.smithi111.stdout: osd debug op order = true 2023-10-03T17:34:12.006 INFO:teuthology.orchestra.run.smithi111.stdout: osd debug verify stray on activate = true 2023-10-03T17:34:12.006 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-03T17:34:12.006 INFO:teuthology.orchestra.run.smithi111.stdout: osd open classes on start = true 2023-10-03T17:34:12.007 INFO:teuthology.orchestra.run.smithi111.stdout: osd debug pg log writeout = true 2023-10-03T17:34:12.007 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-03T17:34:12.007 INFO:teuthology.orchestra.run.smithi111.stdout: osd deep scrub update digest min age = 30 2023-10-03T17:34:12.007 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-03T17:34:12.007 INFO:teuthology.orchestra.run.smithi111.stdout: osd map max advance = 10 2023-10-03T17:34:12.007 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-03T17:34:12.007 INFO:teuthology.orchestra.run.smithi111.stdout: journal zero on create = true 2023-10-03T17:34:12.007 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-03T17:34:12.007 INFO:teuthology.orchestra.run.smithi111.stdout: filestore ondisk finisher threads = 3 2023-10-03T17:34:12.007 INFO:teuthology.orchestra.run.smithi111.stdout: filestore apply finisher threads = 3 2023-10-03T17:34:12.007 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-03T17:34:12.008 INFO:teuthology.orchestra.run.smithi111.stdout: bdev debug aio = true 2023-10-03T17:34:12.008 INFO:teuthology.orchestra.run.smithi111.stdout: osd debug misdirected ops = true 2023-10-03T17:34:12.008 INFO:teuthology.orchestra.run.smithi111.stdout: bluestore block size = 96636764160 2023-10-03T17:34:12.008 INFO:teuthology.orchestra.run.smithi111.stdout: bluestore compression algorithm = zlib 2023-10-03T17:34:12.008 INFO:teuthology.orchestra.run.smithi111.stdout: bluestore compression mode = aggressive 2023-10-03T17:34:12.008 INFO:teuthology.orchestra.run.smithi111.stdout: bluestore fsck on mount = True 2023-10-03T17:34:12.008 INFO:teuthology.orchestra.run.smithi111.stdout: debug bluefs = 1/20 2023-10-03T17:34:12.008 INFO:teuthology.orchestra.run.smithi111.stdout: debug bluestore = 1/20 2023-10-03T17:34:12.008 INFO:teuthology.orchestra.run.smithi111.stdout: debug ms = 1 2023-10-03T17:34:12.008 INFO:teuthology.orchestra.run.smithi111.stdout: debug osd = 20 2023-10-03T17:34:12.009 INFO:teuthology.orchestra.run.smithi111.stdout: debug rocksdb = 4/10 2023-10-03T17:34:12.009 INFO:teuthology.orchestra.run.smithi111.stdout: mon osd backfillfull_ratio = 0.85 2023-10-03T17:34:12.009 INFO:teuthology.orchestra.run.smithi111.stdout: mon osd full ratio = 0.9 2023-10-03T17:34:12.009 INFO:teuthology.orchestra.run.smithi111.stdout: mon osd nearfull ratio = 0.8 2023-10-03T17:34:12.009 INFO:teuthology.orchestra.run.smithi111.stdout: osd failsafe full ratio = 0.95 2023-10-03T17:34:12.009 INFO:teuthology.orchestra.run.smithi111.stdout: osd objectstore = bluestore 2023-10-03T17:34:12.009 INFO:teuthology.orchestra.run.smithi111.stdout: osd shutdown pgref assert = True 2023-10-03T17:34:12.009 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-03T17:34:12.010 INFO:teuthology.orchestra.run.smithi111.stdout:[mgr] 2023-10-03T17:34:12.010 INFO:teuthology.orchestra.run.smithi111.stdout: debug ms = 1 2023-10-03T17:34:12.010 INFO:teuthology.orchestra.run.smithi111.stdout: debug mgr = 20 2023-10-03T17:34:12.010 INFO:teuthology.orchestra.run.smithi111.stdout: debug mon = 20 2023-10-03T17:34:12.010 INFO:teuthology.orchestra.run.smithi111.stdout: debug auth = 20 2023-10-03T17:34:12.010 INFO:teuthology.orchestra.run.smithi111.stdout: mon reweight min pgs per osd = 4 2023-10-03T17:34:12.010 INFO:teuthology.orchestra.run.smithi111.stdout: mon reweight min bytes per osd = 10 2023-10-03T17:34:12.010 INFO:teuthology.orchestra.run.smithi111.stdout: mgr/telemetry/nag = false 2023-10-03T17:34:12.010 INFO:teuthology.orchestra.run.smithi111.stdout: debug rbd = 20 2023-10-03T17:34:12.010 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-03T17:34:12.011 INFO:teuthology.orchestra.run.smithi111.stdout:[mon] 2023-10-03T17:34:12.011 INFO:teuthology.orchestra.run.smithi111.stdout: debug ms = 1 2023-10-03T17:34:12.011 INFO:teuthology.orchestra.run.smithi111.stdout: debug mon = 20 2023-10-03T17:34:12.011 INFO:teuthology.orchestra.run.smithi111.stdout: debug paxos = 20 2023-10-03T17:34:12.011 INFO:teuthology.orchestra.run.smithi111.stdout: debug auth = 20 2023-10-03T17:34:12.011 INFO:teuthology.orchestra.run.smithi111.stdout: mon data avail warn = 5 2023-10-03T17:34:12.011 INFO:teuthology.orchestra.run.smithi111.stdout: mon mgr mkfs grace = 240 2023-10-03T17:34:12.011 INFO:teuthology.orchestra.run.smithi111.stdout: mon reweight min pgs per osd = 4 2023-10-03T17:34:12.011 INFO:teuthology.orchestra.run.smithi111.stdout: mon osd reporter subtree level = osd 2023-10-03T17:34:12.011 INFO:teuthology.orchestra.run.smithi111.stdout: mon osd prime pg temp = true 2023-10-03T17:34:12.012 INFO:teuthology.orchestra.run.smithi111.stdout: mon reweight min bytes per osd = 10 2023-10-03T17:34:12.012 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-03T17:34:12.012 INFO:teuthology.orchestra.run.smithi111.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-10-03T17:34:12.012 INFO:teuthology.orchestra.run.smithi111.stdout: auth mon ticket ttl = 660 # 11m 2023-10-03T17:34:12.012 INFO:teuthology.orchestra.run.smithi111.stdout: auth service ticket ttl = 240 # 4m 2023-10-03T17:34:12.012 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-03T17:34:12.012 INFO:teuthology.orchestra.run.smithi111.stdout: # don't complain about insecure global_id in the test suite 2023-10-03T17:34:12.012 INFO:teuthology.orchestra.run.smithi111.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-10-03T17:34:12.012 INFO:teuthology.orchestra.run.smithi111.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-10-03T17:34:12.012 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-03T17:34:12.012 INFO:teuthology.orchestra.run.smithi111.stdout: # 1m isn't quite enough 2023-10-03T17:34:12.013 INFO:teuthology.orchestra.run.smithi111.stdout: mon_down_mkfs_grace = 2m 2023-10-03T17:34:12.013 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-03T17:34:12.013 INFO:teuthology.orchestra.run.smithi111.stdout: mon_warn_on_filestore_osds = false 2023-10-03T17:34:12.013 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-03T17:34:12.013 INFO:teuthology.orchestra.run.smithi111.stdout:[client] 2023-10-03T17:34:12.013 INFO:teuthology.orchestra.run.smithi111.stdout: rgw cache enabled = true 2023-10-03T17:34:12.013 INFO:teuthology.orchestra.run.smithi111.stdout: rgw enable ops log = true 2023-10-03T17:34:12.013 INFO:teuthology.orchestra.run.smithi111.stdout: rgw enable usage log = true 2023-10-03T17:34:12.013 INFO:teuthology.orchestra.run.smithi111.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-10-03T17:34:12.013 INFO:teuthology.orchestra.run.smithi111.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-10-03T17:34:12.014 INFO:teuthology.orchestra.run.smithi111.stdout: rbd default data pool = datapool 2023-10-03T17:34:12.014 INFO:teuthology.orchestra.run.smithi111.stdout: rbd default features = 1 2023-10-03T17:34:12.014 INFO:teuthology.orchestra.run.smithi111.stdout:[mon.a] 2023-10-03T17:34:12.015 INFO:tasks.ceph:Creating admin key on mon.a... 2023-10-03T17:34:12.015 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2023-10-03T17:34:12.107 INFO:tasks.ceph:Copying monmap to all nodes... 2023-10-03T17:34:12.107 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-10-03T17:34:12.107 DEBUG:teuthology.orchestra.run.smithi111:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2023-10-03T17:34:12.113 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-10-03T17:34:12.113 DEBUG:teuthology.orchestra.run.smithi111:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2023-10-03T17:34:12.159 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi111.front.sepia.ceph.com 2023-10-03T17:34:12.159 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-10-03T17:34:12.159 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd of=/etc/ceph/ceph.keyring 2023-10-03T17:34:12.159 DEBUG:teuthology.orchestra.run.smithi111:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-10-03T17:34:12.224 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-10-03T17:34:12.224 DEBUG:teuthology.orchestra.run.smithi111:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-10-03T17:34:12.271 INFO:tasks.ceph:Setting up mon nodes... 2023-10-03T17:34:12.271 INFO:tasks.ceph:Setting up mgr nodes... 2023-10-03T17:34:12.272 DEBUG:teuthology.orchestra.run.smithi111:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2023-10-03T17:34:12.356 INFO:teuthology.orchestra.run.smithi111.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2023-10-03T17:34:12.358 INFO:tasks.ceph:Setting up mds nodes... 2023-10-03T17:34:12.358 INFO:tasks.ceph_client:Setting up client nodes... 2023-10-03T17:34:12.359 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-10-03T17:34:12.434 INFO:teuthology.orchestra.run.smithi111.stdout:creating /etc/ceph/ceph.client.0.keyring 2023-10-03T17:34:12.445 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-10-03T17:34:12.445 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi111.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2023-10-03T17:34:12.445 DEBUG:teuthology.orchestra.run.smithi111:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2023-10-03T17:34:12.497 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-10-03T17:34:12.498 INFO:tasks.ceph:role: osd.0 2023-10-03T17:34:12.498 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi111.front.sepia.ceph.com 2023-10-03T17:34:12.498 DEBUG:teuthology.orchestra.run.smithi111:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-10-03T17:34:12.558 INFO:teuthology.orchestra.run.smithi111.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-10-03T17:34:12.558 INFO:teuthology.orchestra.run.smithi111.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-03T17:34:12.559 INFO:teuthology.orchestra.run.smithi111.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-03T17:34:12.559 INFO:teuthology.orchestra.run.smithi111.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-03T17:34:12.559 INFO:teuthology.orchestra.run.smithi111.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-03T17:34:12.559 INFO:teuthology.orchestra.run.smithi111.stdout: = sunit=0 swidth=0 blks 2023-10-03T17:34:12.559 INFO:teuthology.orchestra.run.smithi111.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-03T17:34:12.559 INFO:teuthology.orchestra.run.smithi111.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-03T17:34:12.559 INFO:teuthology.orchestra.run.smithi111.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-03T17:34:12.559 INFO:teuthology.orchestra.run.smithi111.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-03T17:34:13.009 INFO:teuthology.orchestra.run.smithi111.stdout:Discarding blocks...Done. 2023-10-03T17:34:13.011 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi111.front.sepia.ceph.com -o noatime 2023-10-03T17:34:13.011 DEBUG:teuthology.orchestra.run.smithi111:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2023-10-03T17:34:13.101 DEBUG:teuthology.orchestra.run.smithi111:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2023-10-03T17:34:13.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-03T17:34:13.153 INFO:teuthology.orchestra.run.smithi111.stderr:sudo: /sbin/restorecon: command not found 2023-10-03T17:34:13.154 DEBUG:teuthology.orchestra.run.smithi111:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2023-10-03T17:34:13.206 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-10-03T17:34:13.207 INFO:tasks.ceph:role: osd.1 2023-10-03T17:34:13.207 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi111.front.sepia.ceph.com 2023-10-03T17:34:13.207 DEBUG:teuthology.orchestra.run.smithi111:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-10-03T17:34:13.271 INFO:teuthology.orchestra.run.smithi111.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-10-03T17:34:13.271 INFO:teuthology.orchestra.run.smithi111.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-03T17:34:13.271 INFO:teuthology.orchestra.run.smithi111.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-03T17:34:13.271 INFO:teuthology.orchestra.run.smithi111.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-03T17:34:13.272 INFO:teuthology.orchestra.run.smithi111.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-03T17:34:13.272 INFO:teuthology.orchestra.run.smithi111.stdout: = sunit=0 swidth=0 blks 2023-10-03T17:34:13.272 INFO:teuthology.orchestra.run.smithi111.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-03T17:34:13.272 INFO:teuthology.orchestra.run.smithi111.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-03T17:34:13.272 INFO:teuthology.orchestra.run.smithi111.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-03T17:34:13.272 INFO:teuthology.orchestra.run.smithi111.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-03T17:34:13.729 INFO:teuthology.orchestra.run.smithi111.stdout:Discarding blocks...Done. 2023-10-03T17:34:13.731 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi111.front.sepia.ceph.com -o noatime 2023-10-03T17:34:13.732 DEBUG:teuthology.orchestra.run.smithi111:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2023-10-03T17:34:13.886 DEBUG:teuthology.orchestra.run.smithi111:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2023-10-03T17:34:13.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-03T17:34:13.898 INFO:teuthology.orchestra.run.smithi111.stderr:sudo: /sbin/restorecon: command not found 2023-10-03T17:34:13.898 DEBUG:teuthology.orchestra.run.smithi111:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2023-10-03T17:34:13.951 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-10-03T17:34:13.951 INFO:tasks.ceph:role: osd.2 2023-10-03T17:34:13.951 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi111.front.sepia.ceph.com 2023-10-03T17:34:13.951 DEBUG:teuthology.orchestra.run.smithi111:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-10-03T17:34:14.015 INFO:teuthology.orchestra.run.smithi111.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-10-03T17:34:14.015 INFO:teuthology.orchestra.run.smithi111.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-03T17:34:14.015 INFO:teuthology.orchestra.run.smithi111.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-03T17:34:14.015 INFO:teuthology.orchestra.run.smithi111.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-03T17:34:14.015 INFO:teuthology.orchestra.run.smithi111.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-03T17:34:14.015 INFO:teuthology.orchestra.run.smithi111.stdout: = sunit=0 swidth=0 blks 2023-10-03T17:34:14.015 INFO:teuthology.orchestra.run.smithi111.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-03T17:34:14.016 INFO:teuthology.orchestra.run.smithi111.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-03T17:34:14.016 INFO:teuthology.orchestra.run.smithi111.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-03T17:34:14.016 INFO:teuthology.orchestra.run.smithi111.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-03T17:34:14.482 INFO:teuthology.orchestra.run.smithi111.stdout:Discarding blocks...Done. 2023-10-03T17:34:14.484 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi111.front.sepia.ceph.com -o noatime 2023-10-03T17:34:14.484 DEBUG:teuthology.orchestra.run.smithi111:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2023-10-03T17:34:14.617 DEBUG:teuthology.orchestra.run.smithi111:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2023-10-03T17:34:14.627 INFO:teuthology.orchestra.run.smithi111.stderr:sudo: /sbin/restorecon: command not found 2023-10-03T17:34:14.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-03T17:34:14.628 DEBUG:teuthology.orchestra.run.smithi111:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-03T17:34:14.761 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-03T17:34:14.760+0000 7fbc9d760640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2023-10-03T17:34:14.761 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-03T17:34:14.760+0000 7fbc9d760640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2023-10-03T17:34:14.762 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-03T17:34:14.760+0000 7fbc9d760640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-10-03T17:34:14.762 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-03T17:34:14.760+0000 7fbc9d760640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-10-03T17:34:14.762 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-03T17:34:14.760+0000 7fbc9d760640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-10-03T17:34:14.762 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-03T17:34:14.760+0000 7fbc9d760640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2023-10-03T17:34:17.772 DEBUG:teuthology.orchestra.run.smithi111:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-10-03T17:34:17.785 DEBUG:teuthology.orchestra.run.smithi111:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-03T17:34:17.866 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-03T17:34:17.864+0000 7f26dbf6b640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2023-10-03T17:34:17.867 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-03T17:34:17.868+0000 7f26dbf6b640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2023-10-03T17:34:17.867 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-03T17:34:17.868+0000 7f26dbf6b640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-10-03T17:34:17.867 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-03T17:34:17.868+0000 7f26dbf6b640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-10-03T17:34:17.867 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-03T17:34:17.868+0000 7f26dbf6b640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-10-03T17:34:17.867 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-03T17:34:17.868+0000 7f26dbf6b640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2023-10-03T17:34:20.891 DEBUG:teuthology.orchestra.run.smithi111:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-10-03T17:34:20.948 DEBUG:teuthology.orchestra.run.smithi111:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-03T17:34:21.030 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-03T17:34:21.028+0000 7f1e435fc640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2023-10-03T17:34:21.031 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-03T17:34:21.032+0000 7f1e435fc640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2023-10-03T17:34:21.031 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-03T17:34:21.032+0000 7f1e435fc640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-10-03T17:34:21.032 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-03T17:34:21.032+0000 7f1e435fc640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-10-03T17:34:21.032 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-03T17:34:21.032+0000 7f1e435fc640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-10-03T17:34:21.032 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-03T17:34:21.032+0000 7f1e435fc640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2023-10-03T17:34:24.037 DEBUG:teuthology.orchestra.run.smithi111:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-10-03T17:34:24.092 INFO:tasks.ceph:Reading keys from all nodes... 2023-10-03T17:34:24.092 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-10-03T17:34:24.093 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2023-10-03T17:34:24.146 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-10-03T17:34:24.146 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2023-10-03T17:34:24.198 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-10-03T17:34:24.198 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2023-10-03T17:34:24.254 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-10-03T17:34:24.255 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2023-10-03T17:34:24.310 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-10-03T17:34:24.310 DEBUG:teuthology.orchestra.run.smithi111:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2023-10-03T17:34:24.362 INFO:tasks.ceph:Adding keys to all mons... 2023-10-03T17:34:24.362 DEBUG:teuthology.orchestra.run.smithi111:> sudo tee -a /etc/ceph/ceph.keyring 2023-10-03T17:34:24.417 INFO:teuthology.orchestra.run.smithi111.stdout:[mgr.x] 2023-10-03T17:34:24.418 INFO:teuthology.orchestra.run.smithi111.stdout: key = AQCUUBxlK1RhFRAA1HOwpzLCf5uJ2lAh/FWkBw== 2023-10-03T17:34:24.418 INFO:teuthology.orchestra.run.smithi111.stdout:[osd.0] 2023-10-03T17:34:24.418 INFO:teuthology.orchestra.run.smithi111.stdout: key = AQCWUBxlfPSELRAAMMx0WGsAVxWuEeNPYZDThg== 2023-10-03T17:34:24.418 INFO:teuthology.orchestra.run.smithi111.stdout:[osd.1] 2023-10-03T17:34:24.418 INFO:teuthology.orchestra.run.smithi111.stdout: key = AQCZUBxlaqzLMxAAtjOFCI4awNyCHG8VJ4I+ew== 2023-10-03T17:34:24.418 INFO:teuthology.orchestra.run.smithi111.stdout:[osd.2] 2023-10-03T17:34:24.418 INFO:teuthology.orchestra.run.smithi111.stdout: key = AQCdUBxlYeX1ARAAKRqKQpf10QLQOpVxxuYyJw== 2023-10-03T17:34:24.418 INFO:teuthology.orchestra.run.smithi111.stdout:[client.0] 2023-10-03T17:34:24.418 INFO:teuthology.orchestra.run.smithi111.stdout: key = AQCUUBxljJsOGhAAcWqfv0RDBltFP+KLjRxa8g== 2023-10-03T17:34:24.419 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-10-03T17:34:24.507 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-03T17:34:24.598 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-03T17:34:24.687 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-03T17:34:24.731 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-03T17:34:24.819 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-10-03T17:34:24.820 DEBUG:teuthology.orchestra.run.smithi111:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2023-10-03T17:34:24.830 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-10-03T17:34:24.921 DEBUG:teuthology.orchestra.run.smithi111:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2023-10-03T17:34:24.931 DEBUG:teuthology.orchestra.run.smithi111:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-10-03T17:34:24.978 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2023-10-03T17:34:24.979 INFO:tasks.ceph.mon.a:Restarting daemon 2023-10-03T17:34:24.979 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-10-03T17:34:25.024 INFO:tasks.ceph.mon.a:Started 2023-10-03T17:34:25.024 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2023-10-03T17:34:25.024 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-10-03T17:34:25.024 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2023-10-03T17:34:25.027 INFO:tasks.ceph.mgr.x:Started 2023-10-03T17:34:25.028 INFO:tasks.ceph:Setting crush tunables to default 2023-10-03T17:34:25.028 DEBUG:teuthology.orchestra.run.smithi111:> sudo ceph --cluster ceph osd crush tunables default 2023-10-03T17:34:25.261 INFO:teuthology.orchestra.run.smithi111.stderr:adjusted tunables profile to default 2023-10-03T17:34:25.278 INFO:tasks.ceph:check_enable_crimson: False 2023-10-03T17:34:25.278 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2023-10-03T17:34:25.279 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-10-03T17:34:25.279 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2023-10-03T17:34:25.288 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-10-03T17:34:25.289 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2023-10-03T17:34:25.340 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-10-03T17:34:25.340 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2023-10-03T17:34:25.395 DEBUG:teuthology.orchestra.run.smithi111:> sudo ceph --cluster ceph osd new 7f9ffbc3-1e8c-44e2-8305-abc2b594dbd6 0 2023-10-03T17:34:25.438 INFO:tasks.ceph.mgr.x.smithi111.stderr:2023-10-03T17:34:25.436+0000 7fe4e9b33180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-03T17:34:25.638 INFO:teuthology.orchestra.run.smithi111.stdout:0 2023-10-03T17:34:25.656 DEBUG:teuthology.orchestra.run.smithi111:> sudo ceph --cluster ceph osd new 92793af8-c508-44bb-918d-7a8a92862c82 1 2023-10-03T17:34:25.849 INFO:teuthology.orchestra.run.smithi111.stdout:1 2023-10-03T17:34:25.867 DEBUG:teuthology.orchestra.run.smithi111:> sudo ceph --cluster ceph osd new 354eb2cb-387e-4416-a5b0-c80ffd0bd292 2 2023-10-03T17:34:25.997 INFO:tasks.ceph.mgr.x.smithi111.stderr:2023-10-03T17:34:25.996+0000 7fe4e9b33180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-03T17:34:26.067 INFO:teuthology.orchestra.run.smithi111.stdout:2 2023-10-03T17:34:26.083 INFO:tasks.ceph.osd.0:Restarting daemon 2023-10-03T17:34:26.084 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2023-10-03T17:34:26.085 INFO:tasks.ceph.osd.0:Started 2023-10-03T17:34:26.085 INFO:tasks.ceph.osd.1:Restarting daemon 2023-10-03T17:34:26.086 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2023-10-03T17:34:26.087 INFO:tasks.ceph.osd.1:Started 2023-10-03T17:34:26.087 INFO:tasks.ceph.osd.2:Restarting daemon 2023-10-03T17:34:26.088 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2023-10-03T17:34:26.090 INFO:tasks.ceph.osd.2:Started 2023-10-03T17:34:26.090 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-03T17:34:26.187 INFO:tasks.ceph.mgr.x.smithi111.stderr:2023-10-03T17:34:26.188+0000 7fe4e9b33180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-03T17:34:26.280 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-03T17:34:26.280 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":5,"fsid":"499d6abd-e417-4d8d-a4aa-a7db52cf5eb1","created":"2023-10-03T17:34:25.157173+0000","modified":"2023-10-03T17:34:26.068746+0000","last_up_change":"0.000000","last_in_change":"2023-10-03T17:34:26.068746+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7f9ffbc3-1e8c-44e2-8305-abc2b594dbd6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"92793af8-c508-44bb-918d-7a8a92862c82","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"354eb2cb-387e-4416-a5b0-c80ffd0bd292","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-03T17:34:26.296 INFO:tasks.ceph.ceph_manager.ceph:[] 2023-10-03T17:34:26.296 INFO:tasks.ceph:Waiting for OSDs to come up 2023-10-03T17:34:26.476 INFO:tasks.ceph.osd.2.smithi111.stderr:2023-10-03T17:34:26.476+0000 7f8be48bb640 -1 Falling back to public interface 2023-10-03T17:34:26.598 DEBUG:teuthology.orchestra.run.smithi111:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-03T17:34:26.732 INFO:tasks.ceph.osd.0.smithi111.stderr:2023-10-03T17:34:26.732+0000 7ffbbc899640 -1 Falling back to public interface 2023-10-03T17:34:26.735 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:34:26.736+0000 7f09f4e61640 -1 Falling back to public interface 2023-10-03T17:34:26.760 INFO:tasks.ceph.mgr.x.smithi111.stderr:2023-10-03T17:34:26.760+0000 7fe4e9b33180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-03T17:34:26.778 INFO:teuthology.misc.health.smithi111.stdout: 2023-10-03T17:34:26.778 INFO:teuthology.misc.health.smithi111.stdout:{"epoch":5,"fsid":"499d6abd-e417-4d8d-a4aa-a7db52cf5eb1","created":"2023-10-03T17:34:25.157173+0000","modified":"2023-10-03T17:34:26.068746+0000","last_up_change":"0.000000","last_in_change":"2023-10-03T17:34:26.068746+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7f9ffbc3-1e8c-44e2-8305-abc2b594dbd6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"92793af8-c508-44bb-918d-7a8a92862c82","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"354eb2cb-387e-4416-a5b0-c80ffd0bd292","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-03T17:34:26.792 DEBUG:teuthology.misc:0 of 3 OSDs are up 2023-10-03T17:34:26.816 INFO:tasks.ceph.mgr.x.smithi111.stderr:2023-10-03T17:34:26.816+0000 7fe4e9b33180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-03T17:34:26.868 INFO:tasks.ceph.mgr.x.smithi111.stderr:2023-10-03T17:34:26.868+0000 7fe4e9b33180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-03T17:34:26.945 INFO:tasks.ceph.mgr.x.smithi111.stderr:2023-10-03T17:34:26.944+0000 7fe4e9b33180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-03T17:34:26.996 INFO:tasks.ceph.mgr.x.smithi111.stderr:2023-10-03T17:34:26.996+0000 7fe4e9b33180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-03T17:34:27.073 INFO:tasks.ceph.mgr.x.smithi111.stderr:2023-10-03T17:34:27.072+0000 7fe4e9b33180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-03T17:34:27.124 INFO:tasks.ceph.mgr.x.smithi111.stderr:2023-10-03T17:34:27.124+0000 7fe4e9b33180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-03T17:34:27.262 INFO:tasks.ceph.mgr.x.smithi111.stderr:2023-10-03T17:34:27.260+0000 7fe4e9b33180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-03T17:34:27.321 INFO:tasks.ceph.mgr.x.smithi111.stderr:2023-10-03T17:34:27.320+0000 7fe4e9b33180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-03T17:34:27.982 INFO:tasks.ceph.mgr.x.smithi111.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2023-10-03T17:34:27.982 INFO:tasks.ceph.mgr.x.smithi111.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2023-10-03T17:34:27.982 INFO:tasks.ceph.mgr.x.smithi111.stderr: from numpy import show_config as show_numpy_config 2023-10-03T17:34:28.057 INFO:tasks.ceph.mgr.x.smithi111.stderr:2023-10-03T17:34:28.056+0000 7fe4e9b33180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-03T17:34:28.231 INFO:tasks.ceph.mgr.x.smithi111.stderr:2023-10-03T17:34:28.228+0000 7fe4e9b33180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-03T17:34:28.395 INFO:tasks.ceph.mgr.x.smithi111.stderr:2023-10-03T17:34:28.396+0000 7fe4e9b33180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-03T17:34:28.460 INFO:tasks.ceph.mgr.x.smithi111.stderr:2023-10-03T17:34:28.460+0000 7fe4e9b33180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-03T17:34:28.583 INFO:tasks.ceph.mgr.x.smithi111.stderr:2023-10-03T17:34:28.584+0000 7fe4e9b33180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-03T17:34:29.203 INFO:tasks.ceph.mgr.x.smithi111.stderr:2023-10-03T17:34:29.200+0000 7fe4e9b33180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-03T17:34:29.292 INFO:tasks.ceph.mgr.x.smithi111.stderr:2023-10-03T17:34:29.292+0000 7fe4e9b33180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-03T17:34:29.464 INFO:tasks.ceph.mgr.x.smithi111.stderr:2023-10-03T17:34:29.464+0000 7fe4e9b33180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-03T17:34:29.538 INFO:tasks.ceph.mgr.x.smithi111.stderr:2023-10-03T17:34:29.536+0000 7fe4e9b33180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-03T17:34:29.865 INFO:tasks.ceph.mgr.x.smithi111.stderr:2023-10-03T17:34:29.864+0000 7fe4e9b33180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-03T17:34:29.924 INFO:tasks.ceph.mgr.x.smithi111.stderr:2023-10-03T17:34:29.924+0000 7fe4e9b33180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-03T17:34:30.384 INFO:tasks.ceph.osd.0.smithi111.stderr:2023-10-03T17:34:30.384+0000 7ffbbc899640 -1 osd.0 0 log_to_monitors true 2023-10-03T17:34:30.387 INFO:tasks.ceph.osd.2.smithi111.stderr:2023-10-03T17:34:30.388+0000 7f8be48bb640 -1 osd.2 0 log_to_monitors true 2023-10-03T17:34:30.585 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:34:30.584+0000 7f09f4e61640 -1 osd.1 0 log_to_monitors true 2023-10-03T17:34:30.961 INFO:tasks.ceph.mgr.x.smithi111.stderr:2023-10-03T17:34:30.960+0000 7fe4913a2640 -1 mgr.server handle_report got status from non-daemon mon.a 2023-10-03T17:34:33.094 DEBUG:teuthology.orchestra.run.smithi111:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-03T17:34:33.385 INFO:teuthology.misc.health.smithi111.stdout: 2023-10-03T17:34:33.386 INFO:teuthology.misc.health.smithi111.stdout:{"epoch":7,"fsid":"499d6abd-e417-4d8d-a4aa-a7db52cf5eb1","created":"2023-10-03T17:34:25.157173+0000","modified":"2023-10-03T17:34:32.108063+0000","last_up_change":"0.000000","last_in_change":"2023-10-03T17:34:26.068746+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7f9ffbc3-1e8c-44e2-8305-abc2b594dbd6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"92793af8-c508-44bb-918d-7a8a92862c82","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"354eb2cb-387e-4416-a5b0-c80ffd0bd292","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-03T17:34:33.403 DEBUG:teuthology.misc:0 of 3 OSDs are up 2023-10-03T17:34:35.183 INFO:tasks.ceph.osd.0.smithi111.stderr:2023-10-03T17:34:35.184+0000 7ffbb8841640 -1 osd.0 0 waiting for initial osdmap 2023-10-03T17:34:35.194 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:34:35.192+0000 7f09f0e09640 -1 osd.1 0 waiting for initial osdmap 2023-10-03T17:34:35.211 INFO:tasks.ceph.osd.0.smithi111.stderr:2023-10-03T17:34:35.212+0000 7ffbb0676640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-03T17:34:35.218 INFO:tasks.ceph.osd.2.smithi111.stderr:2023-10-03T17:34:35.216+0000 7f8be1077640 -1 osd.2 0 waiting for initial osdmap 2023-10-03T17:34:35.223 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:34:35.224+0000 7f09e8c3e640 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-03T17:34:35.255 INFO:tasks.ceph.osd.2.smithi111.stderr:2023-10-03T17:34:35.256+0000 7f8bd8698640 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-03T17:34:39.705 DEBUG:teuthology.orchestra.run.smithi111:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-03T17:34:39.993 INFO:teuthology.misc.health.smithi111.stdout: 2023-10-03T17:34:39.993 INFO:teuthology.misc.health.smithi111.stdout:{"epoch":11,"fsid":"499d6abd-e417-4d8d-a4aa-a7db52cf5eb1","created":"2023-10-03T17:34:25.157173+0000","modified":"2023-10-03T17:34:39.232224+0000","last_up_change":"2023-10-03T17:34:36.222314+0000","last_in_change":"2023-10-03T17:34:26.068746+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-03T17:34:38.029646+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7f9ffbc3-1e8c-44e2-8305-abc2b594dbd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":1646836731},{"type":"v1","addr":"172.21.15.111:6809","nonce":1646836731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":1646836731},{"type":"v1","addr":"172.21.15.111:6811","nonce":1646836731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":1646836731},{"type":"v1","addr":"172.21.15.111:6815","nonce":1646836731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":1646836731},{"type":"v1","addr":"172.21.15.111:6813","nonce":1646836731}]},"public_addr":"172.21.15.111:6809/1646836731","cluster_addr":"172.21.15.111:6811/1646836731","heartbeat_back_addr":"172.21.15.111:6815/1646836731","heartbeat_front_addr":"172.21.15.111:6813/1646836731","state":["exists","up"]},{"osd":1,"uuid":"92793af8-c508-44bb-918d-7a8a92862c82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":3440296436},{"type":"v1","addr":"172.21.15.111:6817","nonce":3440296436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":3440296436},{"type":"v1","addr":"172.21.15.111:6819","nonce":3440296436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":3440296436},{"type":"v1","addr":"172.21.15.111:6823","nonce":3440296436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":3440296436},{"type":"v1","addr":"172.21.15.111:6821","nonce":3440296436}]},"public_addr":"172.21.15.111:6817/3440296436","cluster_addr":"172.21.15.111:6819/3440296436","heartbeat_back_addr":"172.21.15.111:6823/3440296436","heartbeat_front_addr":"172.21.15.111:6821/3440296436","state":["exists","up"]},{"osd":2,"uuid":"354eb2cb-387e-4416-a5b0-c80ffd0bd292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":653857722},{"type":"v1","addr":"172.21.15.111:6801","nonce":653857722}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":653857722},{"type":"v1","addr":"172.21.15.111:6803","nonce":653857722}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":653857722},{"type":"v1","addr":"172.21.15.111:6807","nonce":653857722}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":653857722},{"type":"v1","addr":"172.21.15.111:6805","nonce":653857722}]},"public_addr":"172.21.15.111:6801/653857722","cluster_addr":"172.21.15.111:6803/653857722","heartbeat_back_addr":"172.21.15.111:6807/653857722","heartbeat_front_addr":"172.21.15.111:6805/653857722","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-03T17:34:31.339197+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-03T17:34:31.634807+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-03T17:34:31.432310+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-03T17:34:40.011 DEBUG:teuthology.misc:3 of 3 OSDs are up 2023-10-03T17:34:40.011 INFO:tasks.ceph:Creating RBD pool 2023-10-03T17:34:40.011 DEBUG:teuthology.orchestra.run.smithi111:> sudo ceph --cluster ceph osd pool create rbd 8 2023-10-03T17:34:41.243 INFO:teuthology.orchestra.run.smithi111.stderr:pool 'rbd' created 2023-10-03T17:34:41.272 DEBUG:teuthology.orchestra.run.smithi111:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2023-10-03T17:34:42.251 INFO:teuthology.orchestra.run.smithi111.stderr:enabled application 'rbd' on pool 'rbd' 2023-10-03T17:34:42.286 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2023-10-03T17:34:42.286 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-10-03T17:34:42.287 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2023-10-03T17:34:42.287 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-10-03T17:34:42.287 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-10-03T17:34:42.618 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-03T17:34:42.636 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":5,"active_gid":4099,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":1522160706},{"type":"v1","addr":"172.21.15.111:6825","nonce":1522160706}]},"active_addr":"172.21.15.111:6825/1522160706","active_change":"2023-10-03T17:34:29.947029+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:0.0.3","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.111:0","nonce":329999448}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.111:0","nonce":628727734}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.111:0","nonce":3950478362}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.111:0","nonce":200509692}]}]} 2023-10-03T17:34:42.639 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2023-10-03T17:34:42.640 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2023-10-03T17:34:42.640 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-03T17:34:42.928 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-03T17:34:42.928 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":14,"fsid":"499d6abd-e417-4d8d-a4aa-a7db52cf5eb1","created":"2023-10-03T17:34:25.157173+0000","modified":"2023-10-03T17:34:42.246060+0000","last_up_change":"2023-10-03T17:34:36.222314+0000","last_in_change":"2023-10-03T17:34:26.068746+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-03T17:34:38.029646+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-03T17:34:40.410622+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7f9ffbc3-1e8c-44e2-8305-abc2b594dbd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":1646836731},{"type":"v1","addr":"172.21.15.111:6809","nonce":1646836731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":1646836731},{"type":"v1","addr":"172.21.15.111:6811","nonce":1646836731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":1646836731},{"type":"v1","addr":"172.21.15.111:6815","nonce":1646836731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":1646836731},{"type":"v1","addr":"172.21.15.111:6813","nonce":1646836731}]},"public_addr":"172.21.15.111:6809/1646836731","cluster_addr":"172.21.15.111:6811/1646836731","heartbeat_back_addr":"172.21.15.111:6815/1646836731","heartbeat_front_addr":"172.21.15.111:6813/1646836731","state":["exists","up"]},{"osd":1,"uuid":"92793af8-c508-44bb-918d-7a8a92862c82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":3440296436},{"type":"v1","addr":"172.21.15.111:6817","nonce":3440296436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":3440296436},{"type":"v1","addr":"172.21.15.111:6819","nonce":3440296436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":3440296436},{"type":"v1","addr":"172.21.15.111:6823","nonce":3440296436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":3440296436},{"type":"v1","addr":"172.21.15.111:6821","nonce":3440296436}]},"public_addr":"172.21.15.111:6817/3440296436","cluster_addr":"172.21.15.111:6819/3440296436","heartbeat_back_addr":"172.21.15.111:6823/3440296436","heartbeat_front_addr":"172.21.15.111:6821/3440296436","state":["exists","up"]},{"osd":2,"uuid":"354eb2cb-387e-4416-a5b0-c80ffd0bd292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":653857722},{"type":"v1","addr":"172.21.15.111:6801","nonce":653857722}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":653857722},{"type":"v1","addr":"172.21.15.111:6803","nonce":653857722}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":653857722},{"type":"v1","addr":"172.21.15.111:6807","nonce":653857722}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":653857722},{"type":"v1","addr":"172.21.15.111:6805","nonce":653857722}]},"public_addr":"172.21.15.111:6801/653857722","cluster_addr":"172.21.15.111:6803/653857722","heartbeat_back_addr":"172.21.15.111:6807/653857722","heartbeat_front_addr":"172.21.15.111:6805/653857722","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-03T17:34:31.339197+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-03T17:34:31.634807+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-03T17:34:31.432310+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-03T17:34:42.946 INFO:tasks.ceph.ceph_manager.ceph:all up! 2023-10-03T17:34:42.946 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-03T17:34:43.230 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-03T17:34:43.231 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":14,"fsid":"499d6abd-e417-4d8d-a4aa-a7db52cf5eb1","created":"2023-10-03T17:34:25.157173+0000","modified":"2023-10-03T17:34:42.246060+0000","last_up_change":"2023-10-03T17:34:36.222314+0000","last_in_change":"2023-10-03T17:34:26.068746+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-03T17:34:38.029646+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-03T17:34:40.410622+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7f9ffbc3-1e8c-44e2-8305-abc2b594dbd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":1646836731},{"type":"v1","addr":"172.21.15.111:6809","nonce":1646836731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":1646836731},{"type":"v1","addr":"172.21.15.111:6811","nonce":1646836731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":1646836731},{"type":"v1","addr":"172.21.15.111:6815","nonce":1646836731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":1646836731},{"type":"v1","addr":"172.21.15.111:6813","nonce":1646836731}]},"public_addr":"172.21.15.111:6809/1646836731","cluster_addr":"172.21.15.111:6811/1646836731","heartbeat_back_addr":"172.21.15.111:6815/1646836731","heartbeat_front_addr":"172.21.15.111:6813/1646836731","state":["exists","up"]},{"osd":1,"uuid":"92793af8-c508-44bb-918d-7a8a92862c82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":3440296436},{"type":"v1","addr":"172.21.15.111:6817","nonce":3440296436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":3440296436},{"type":"v1","addr":"172.21.15.111:6819","nonce":3440296436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":3440296436},{"type":"v1","addr":"172.21.15.111:6823","nonce":3440296436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":3440296436},{"type":"v1","addr":"172.21.15.111:6821","nonce":3440296436}]},"public_addr":"172.21.15.111:6817/3440296436","cluster_addr":"172.21.15.111:6819/3440296436","heartbeat_back_addr":"172.21.15.111:6823/3440296436","heartbeat_front_addr":"172.21.15.111:6821/3440296436","state":["exists","up"]},{"osd":2,"uuid":"354eb2cb-387e-4416-a5b0-c80ffd0bd292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":653857722},{"type":"v1","addr":"172.21.15.111:6801","nonce":653857722}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":653857722},{"type":"v1","addr":"172.21.15.111:6803","nonce":653857722}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":653857722},{"type":"v1","addr":"172.21.15.111:6807","nonce":653857722}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":653857722},{"type":"v1","addr":"172.21.15.111:6805","nonce":653857722}]},"public_addr":"172.21.15.111:6801/653857722","cluster_addr":"172.21.15.111:6803/653857722","heartbeat_back_addr":"172.21.15.111:6807/653857722","heartbeat_front_addr":"172.21.15.111:6805/653857722","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-03T17:34:31.339197+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-03T17:34:31.634807+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-03T17:34:31.432310+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-03T17:34:43.248 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2023-10-03T17:34:43.249 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2023-10-03T17:34:43.249 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2023-10-03T17:34:43.401 INFO:teuthology.orchestra.run.smithi111.stdout:34359738374 2023-10-03T17:34:43.401 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-10-03T17:34:43.405 INFO:teuthology.orchestra.run.smithi111.stdout:34359738374 2023-10-03T17:34:43.405 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-10-03T17:34:43.410 INFO:teuthology.orchestra.run.smithi111.stdout:34359738375 2023-10-03T17:34:43.411 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-10-03T17:34:43.673 INFO:teuthology.orchestra.run.smithi111.stdout:34359738372 2023-10-03T17:34:43.691 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738374 got 34359738372 for osd.0 2023-10-03T17:34:43.740 INFO:teuthology.orchestra.run.smithi111.stdout:34359738372 2023-10-03T17:34:43.742 INFO:teuthology.orchestra.run.smithi111.stdout:34359738373 2023-10-03T17:34:43.759 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738374 got 34359738372 for osd.1 2023-10-03T17:34:43.760 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738375 got 34359738373 for osd.2 2023-10-03T17:34:44.692 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-10-03T17:34:44.761 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-10-03T17:34:44.761 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-10-03T17:34:44.991 INFO:teuthology.orchestra.run.smithi111.stdout:34359738374 2023-10-03T17:34:45.008 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738374 got 34359738374 for osd.0 2023-10-03T17:34:45.009 DEBUG:teuthology.parallel:result is None 2023-10-03T17:34:45.035 INFO:teuthology.orchestra.run.smithi111.stdout:34359738375 2023-10-03T17:34:45.044 INFO:teuthology.orchestra.run.smithi111.stdout:34359738374 2023-10-03T17:34:45.054 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738375 got 34359738375 for osd.2 2023-10-03T17:34:45.054 DEBUG:teuthology.parallel:result is None 2023-10-03T17:34:45.058 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738374 got 34359738374 for osd.1 2023-10-03T17:34:45.059 DEBUG:teuthology.parallel:result is None 2023-10-03T17:34:45.059 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-10-03T17:34:45.059 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T17:34:45.367 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-03T17:34:45.367 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-10-03T17:34:45.383 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":17,"stamp":"2023-10-03T17:34:43.957452+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":7,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":81564,"kb_used_data":472,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":283033956,"statfs":{"total":289910292480,"available":289826770944,"internally_reserved":0,"allocated":483328,"data_stored":1281896,"data_compressed":7460,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4782,"internal_metadata":82963794},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"1.702808"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-10-03T17:34:43.263294+0000","last_change":"2023-10-03T17:34:42.277381+0000","last_active":"2023-10-03T17:34:43.263294+0000","last_peered":"2023-10-03T17:34:43.263294+0000","last_clean":"2023-10-03T17:34:43.263294+0000","last_became_active":"2023-10-03T17:34:42.276624+0000","last_became_peered":"2023-10-03T17:34:42.276624+0000","last_unstale":"2023-10-03T17:34:43.263294+0000","last_undegraded":"2023-10-03T17:34:43.263294+0000","last_fullsized":"2023-10-03T17:34:43.263294+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:34:41.238033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:34:41.238033+0000","last_clean_scrub_stamp":"2023-10-03T17:34:41.238033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T18:56:53.056366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-10-03T17:34:43.263377+0000","last_change":"2023-10-03T17:34:42.271628+0000","last_active":"2023-10-03T17:34:43.263377+0000","last_peered":"2023-10-03T17:34:43.263377+0000","last_clean":"2023-10-03T17:34:43.263377+0000","last_became_active":"2023-10-03T17:34:42.271201+0000","last_became_peered":"2023-10-03T17:34:42.271201+0000","last_unstale":"2023-10-03T17:34:43.263377+0000","last_undegraded":"2023-10-03T17:34:43.263377+0000","last_fullsized":"2023-10-03T17:34:43.263377+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:34:41.238033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:34:41.238033+0000","last_clean_scrub_stamp":"2023-10-03T17:34:41.238033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:13:49.428992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-10-03T17:34:43.263110+0000","last_change":"2023-10-03T17:34:42.277773+0000","last_active":"2023-10-03T17:34:43.263110+0000","last_peered":"2023-10-03T17:34:43.263110+0000","last_clean":"2023-10-03T17:34:43.263110+0000","last_became_active":"2023-10-03T17:34:42.276941+0000","last_became_peered":"2023-10-03T17:34:42.276941+0000","last_unstale":"2023-10-03T17:34:43.263110+0000","last_undegraded":"2023-10-03T17:34:43.263110+0000","last_fullsized":"2023-10-03T17:34:43.263110+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:34:41.238033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:34:41.238033+0000","last_clean_scrub_stamp":"2023-10-03T17:34:41.238033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T21:07:05.154169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-10-03T17:34:43.263099+0000","last_change":"2023-10-03T17:34:42.276876+0000","last_active":"2023-10-03T17:34:43.263099+0000","last_peered":"2023-10-03T17:34:43.263099+0000","last_clean":"2023-10-03T17:34:43.263099+0000","last_became_active":"2023-10-03T17:34:42.276459+0000","last_became_peered":"2023-10-03T17:34:42.276459+0000","last_unstale":"2023-10-03T17:34:43.263099+0000","last_undegraded":"2023-10-03T17:34:43.263099+0000","last_fullsized":"2023-10-03T17:34:43.263099+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:34:41.238033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:34:41.238033+0000","last_clean_scrub_stamp":"2023-10-03T17:34:41.238033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:03:40.525005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-10-03T17:34:42.271479+0000","last_change":"2023-10-03T17:34:42.271479+0000","last_active":"2023-10-03T17:34:42.271479+0000","last_peered":"2023-10-03T17:34:42.271479+0000","last_clean":"2023-10-03T17:34:42.271479+0000","last_became_active":"2023-10-03T17:34:42.270625+0000","last_became_peered":"2023-10-03T17:34:42.270625+0000","last_unstale":"2023-10-03T17:34:42.271479+0000","last_undegraded":"2023-10-03T17:34:42.271479+0000","last_fullsized":"2023-10-03T17:34:42.271479+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:34:41.238033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:34:41.238033+0000","last_clean_scrub_stamp":"2023-10-03T17:34:41.238033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T18:39:54.291704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-10-03T17:34:42.278925+0000","last_change":"2023-10-03T17:34:42.278925+0000","last_active":"2023-10-03T17:34:42.278925+0000","last_peered":"2023-10-03T17:34:42.278925+0000","last_clean":"2023-10-03T17:34:42.278925+0000","last_became_active":"2023-10-03T17:34:42.278196+0000","last_became_peered":"2023-10-03T17:34:42.278196+0000","last_unstale":"2023-10-03T17:34:42.278925+0000","last_undegraded":"2023-10-03T17:34:42.278925+0000","last_fullsized":"2023-10-03T17:34:42.278925+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:34:41.238033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:34:41.238033+0000","last_clean_scrub_stamp":"2023-10-03T17:34:41.238033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T21:43:43.345205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-10-03T17:34:42.273043+0000","last_change":"2023-10-03T17:34:42.273043+0000","last_active":"2023-10-03T17:34:42.273043+0000","last_peered":"2023-10-03T17:34:42.273043+0000","last_clean":"2023-10-03T17:34:42.273043+0000","last_became_active":"2023-10-03T17:34:42.272306+0000","last_became_peered":"2023-10-03T17:34:42.272306+0000","last_unstale":"2023-10-03T17:34:42.273043+0000","last_undegraded":"2023-10-03T17:34:42.273043+0000","last_fullsized":"2023-10-03T17:34:42.273043+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:34:41.238033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:34:41.238033+0000","last_clean_scrub_stamp":"2023-10-03T17:34:41.238033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:31:57.578976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-10-03T17:34:43.263107+0000","last_change":"2023-10-03T17:34:42.271102+0000","last_active":"2023-10-03T17:34:43.263107+0000","last_peered":"2023-10-03T17:34:43.263107+0000","last_clean":"2023-10-03T17:34:43.263107+0000","last_became_active":"2023-10-03T17:34:42.270194+0000","last_became_peered":"2023-10-03T17:34:42.270194+0000","last_unstale":"2023-10-03T17:34:43.263107+0000","last_undegraded":"2023-10-03T17:34:43.263107+0000","last_fullsized":"2023-10-03T17:34:43.263107+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:34:41.238033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:34:41.238033+0000","last_clean_scrub_stamp":"2023-10-03T17:34:41.238033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:41:26.441632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":141,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-10-03T17:34:42.265135+0000","last_change":"2023-10-03T17:34:39.504748+0000","last_active":"2023-10-03T17:34:42.265135+0000","last_peered":"2023-10-03T17:34:42.265135+0000","last_clean":"2023-10-03T17:34:42.265135+0000","last_became_active":"2023-10-03T17:34:39.502393+0000","last_became_peered":"2023-10-03T17:34:39.502393+0000","last_unstale":"2023-10-03T17:34:42.265135+0000","last_undegraded":"2023-10-03T17:34:42.265135+0000","last_fullsized":"2023-10-03T17:34:42.265135+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:34:38.227481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:34:38.227481+0000","last_clean_scrub_stamp":"2023-10-03T17:34:38.227481+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T00:10:27.940656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":7460,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738375,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27148,"kb_used_data":120,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344692,"statfs":{"total":96636764160,"available":96608964608,"internally_reserved":0,"allocated":122880,"data_stored":33720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60099999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71899999999999997}]}]},{"osd":1,"up_from":8,"seq":34359738374,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27216,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344624,"statfs":{"total":96636764160,"available":96608894976,"internally_reserved":0,"allocated":180224,"data_stored":624088,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79600000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63900000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738374,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27200,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344640,"statfs":{"total":96636764160,"available":96608911360,"internally_reserved":0,"allocated":180224,"data_stored":624088,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36199999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51500000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-03T17:34:45.384 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T17:34:45.649 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-03T17:34:45.649 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-10-03T17:34:45.665 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":17,"stamp":"2023-10-03T17:34:43.957452+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":7,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":81564,"kb_used_data":472,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":283033956,"statfs":{"total":289910292480,"available":289826770944,"internally_reserved":0,"allocated":483328,"data_stored":1281896,"data_compressed":7460,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4782,"internal_metadata":82963794},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"1.702808"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-10-03T17:34:43.263294+0000","last_change":"2023-10-03T17:34:42.277381+0000","last_active":"2023-10-03T17:34:43.263294+0000","last_peered":"2023-10-03T17:34:43.263294+0000","last_clean":"2023-10-03T17:34:43.263294+0000","last_became_active":"2023-10-03T17:34:42.276624+0000","last_became_peered":"2023-10-03T17:34:42.276624+0000","last_unstale":"2023-10-03T17:34:43.263294+0000","last_undegraded":"2023-10-03T17:34:43.263294+0000","last_fullsized":"2023-10-03T17:34:43.263294+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:34:41.238033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:34:41.238033+0000","last_clean_scrub_stamp":"2023-10-03T17:34:41.238033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T18:56:53.056366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-10-03T17:34:43.263377+0000","last_change":"2023-10-03T17:34:42.271628+0000","last_active":"2023-10-03T17:34:43.263377+0000","last_peered":"2023-10-03T17:34:43.263377+0000","last_clean":"2023-10-03T17:34:43.263377+0000","last_became_active":"2023-10-03T17:34:42.271201+0000","last_became_peered":"2023-10-03T17:34:42.271201+0000","last_unstale":"2023-10-03T17:34:43.263377+0000","last_undegraded":"2023-10-03T17:34:43.263377+0000","last_fullsized":"2023-10-03T17:34:43.263377+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:34:41.238033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:34:41.238033+0000","last_clean_scrub_stamp":"2023-10-03T17:34:41.238033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:13:49.428992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-10-03T17:34:43.263110+0000","last_change":"2023-10-03T17:34:42.277773+0000","last_active":"2023-10-03T17:34:43.263110+0000","last_peered":"2023-10-03T17:34:43.263110+0000","last_clean":"2023-10-03T17:34:43.263110+0000","last_became_active":"2023-10-03T17:34:42.276941+0000","last_became_peered":"2023-10-03T17:34:42.276941+0000","last_unstale":"2023-10-03T17:34:43.263110+0000","last_undegraded":"2023-10-03T17:34:43.263110+0000","last_fullsized":"2023-10-03T17:34:43.263110+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:34:41.238033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:34:41.238033+0000","last_clean_scrub_stamp":"2023-10-03T17:34:41.238033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T21:07:05.154169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-10-03T17:34:43.263099+0000","last_change":"2023-10-03T17:34:42.276876+0000","last_active":"2023-10-03T17:34:43.263099+0000","last_peered":"2023-10-03T17:34:43.263099+0000","last_clean":"2023-10-03T17:34:43.263099+0000","last_became_active":"2023-10-03T17:34:42.276459+0000","last_became_peered":"2023-10-03T17:34:42.276459+0000","last_unstale":"2023-10-03T17:34:43.263099+0000","last_undegraded":"2023-10-03T17:34:43.263099+0000","last_fullsized":"2023-10-03T17:34:43.263099+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:34:41.238033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:34:41.238033+0000","last_clean_scrub_stamp":"2023-10-03T17:34:41.238033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:03:40.525005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-10-03T17:34:42.271479+0000","last_change":"2023-10-03T17:34:42.271479+0000","last_active":"2023-10-03T17:34:42.271479+0000","last_peered":"2023-10-03T17:34:42.271479+0000","last_clean":"2023-10-03T17:34:42.271479+0000","last_became_active":"2023-10-03T17:34:42.270625+0000","last_became_peered":"2023-10-03T17:34:42.270625+0000","last_unstale":"2023-10-03T17:34:42.271479+0000","last_undegraded":"2023-10-03T17:34:42.271479+0000","last_fullsized":"2023-10-03T17:34:42.271479+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:34:41.238033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:34:41.238033+0000","last_clean_scrub_stamp":"2023-10-03T17:34:41.238033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T18:39:54.291704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-10-03T17:34:42.278925+0000","last_change":"2023-10-03T17:34:42.278925+0000","last_active":"2023-10-03T17:34:42.278925+0000","last_peered":"2023-10-03T17:34:42.278925+0000","last_clean":"2023-10-03T17:34:42.278925+0000","last_became_active":"2023-10-03T17:34:42.278196+0000","last_became_peered":"2023-10-03T17:34:42.278196+0000","last_unstale":"2023-10-03T17:34:42.278925+0000","last_undegraded":"2023-10-03T17:34:42.278925+0000","last_fullsized":"2023-10-03T17:34:42.278925+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:34:41.238033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:34:41.238033+0000","last_clean_scrub_stamp":"2023-10-03T17:34:41.238033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T21:43:43.345205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-10-03T17:34:42.273043+0000","last_change":"2023-10-03T17:34:42.273043+0000","last_active":"2023-10-03T17:34:42.273043+0000","last_peered":"2023-10-03T17:34:42.273043+0000","last_clean":"2023-10-03T17:34:42.273043+0000","last_became_active":"2023-10-03T17:34:42.272306+0000","last_became_peered":"2023-10-03T17:34:42.272306+0000","last_unstale":"2023-10-03T17:34:42.273043+0000","last_undegraded":"2023-10-03T17:34:42.273043+0000","last_fullsized":"2023-10-03T17:34:42.273043+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:34:41.238033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:34:41.238033+0000","last_clean_scrub_stamp":"2023-10-03T17:34:41.238033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:31:57.578976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-10-03T17:34:43.263107+0000","last_change":"2023-10-03T17:34:42.271102+0000","last_active":"2023-10-03T17:34:43.263107+0000","last_peered":"2023-10-03T17:34:43.263107+0000","last_clean":"2023-10-03T17:34:43.263107+0000","last_became_active":"2023-10-03T17:34:42.270194+0000","last_became_peered":"2023-10-03T17:34:42.270194+0000","last_unstale":"2023-10-03T17:34:43.263107+0000","last_undegraded":"2023-10-03T17:34:43.263107+0000","last_fullsized":"2023-10-03T17:34:43.263107+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:34:41.238033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:34:41.238033+0000","last_clean_scrub_stamp":"2023-10-03T17:34:41.238033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:41:26.441632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":141,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-10-03T17:34:42.265135+0000","last_change":"2023-10-03T17:34:39.504748+0000","last_active":"2023-10-03T17:34:42.265135+0000","last_peered":"2023-10-03T17:34:42.265135+0000","last_clean":"2023-10-03T17:34:42.265135+0000","last_became_active":"2023-10-03T17:34:39.502393+0000","last_became_peered":"2023-10-03T17:34:39.502393+0000","last_unstale":"2023-10-03T17:34:42.265135+0000","last_undegraded":"2023-10-03T17:34:42.265135+0000","last_fullsized":"2023-10-03T17:34:42.265135+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:34:38.227481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:34:38.227481+0000","last_clean_scrub_stamp":"2023-10-03T17:34:38.227481+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T00:10:27.940656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":7460,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738375,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27148,"kb_used_data":120,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344692,"statfs":{"total":96636764160,"available":96608964608,"internally_reserved":0,"allocated":122880,"data_stored":33720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60099999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71899999999999997}]}]},{"osd":1,"up_from":8,"seq":34359738374,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27216,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344624,"statfs":{"total":96636764160,"available":96608894976,"internally_reserved":0,"allocated":180224,"data_stored":624088,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79600000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63900000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738374,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27200,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344640,"statfs":{"total":96636764160,"available":96608911360,"internally_reserved":0,"allocated":180224,"data_stored":624088,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36199999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51500000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-03T17:34:45.666 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-10-03T17:34:45.666 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-10-03T17:34:45.667 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2023-10-03T17:34:45.667 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-10-03T17:34:45.969 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-03T17:34:45.969 INFO:teuthology.orchestra.run.smithi111.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-10-03T17:34:45.986 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2023-10-03T17:34:45.986 INFO:teuthology.run_tasks:Running task exec... 2023-10-03T17:34:45.996 INFO:teuthology.task.exec:Executing custom commands... 2023-10-03T17:34:45.996 INFO:teuthology.task.exec:Running commands on role client.0 host ubuntu@smithi111.front.sepia.ceph.com 2023-10-03T17:34:45.996 DEBUG:teuthology.orchestra.run.smithi111:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph osd pool create datapool 4' 2023-10-03T17:34:47.287 INFO:teuthology.orchestra.run.smithi111.stderr:pool 'datapool' created 2023-10-03T17:34:47.311 DEBUG:teuthology.orchestra.run.smithi111:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'rbd pool init datapool' 2023-10-03T17:34:50.338 INFO:teuthology.run_tasks:Running task install... 2023-10-03T17:34:50.352 DEBUG:teuthology.task.install:project ceph 2023-10-03T17:34:50.352 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'a7de91bfa5778a0d559a7e9bc8e35f93456b41ad'}} 2023-10-03T17:34:50.352 DEBUG:teuthology.task.install:config {'extra_system_packages': ['fio'], 'flavor': 'default', 'sha1': 'a7de91bfa5778a0d559a7e9bc8e35f93456b41ad'} 2023-10-03T17:34:50.353 INFO:teuthology.task.install:Using flavor: default 2023-10-03T17:34:50.359 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-10-03T17:34:50.359 INFO:teuthology.task.install:extra packages: [] 2023-10-03T17:34:50.360 DEBUG:teuthology.orchestra.run.smithi111:> sudo apt-key list | grep Ceph 2023-10-03T17:34:50.421 INFO:teuthology.orchestra.run.smithi111.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2023-10-03T17:34:50.451 INFO:teuthology.orchestra.run.smithi111.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2023-10-03T17:34:50.451 INFO:teuthology.orchestra.run.smithi111.stdout:uid [ unknown] Ceph.com (release key) 2023-10-03T17:34:50.452 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2023-10-03T17:34:50.452 INFO:teuthology.task.install.deb:Installing system (non-project) packages: fio on remote deb x86_64 2023-10-03T17:34:50.452 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=a7de91bfa5778a0d559a7e9bc8e35f93456b41ad 2023-10-03T17:34:50.687 INFO:teuthology.task.install.deb:Pulling from https://4.chacra.ceph.com/r/ceph/wip-ajarr-63009-only-reacquire-kickstart/a7de91bfa5778a0d559a7e9bc8e35f93456b41ad/ubuntu/jammy/flavors/default/ 2023-10-03T17:34:50.688 INFO:teuthology.task.install.deb:Package version is 18.0.0-6336-ga7de91bf-1jammy 2023-10-03T17:34:50.793 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-10-03T17:34:50.793 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2023-10-03T17:34:50.810 DEBUG:teuthology.orchestra.run.smithi111:> sudo apt-get update 2023-10-03T17:34:51.009 INFO:teuthology.orchestra.run.smithi111.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2023-10-03T17:34:51.030 INFO:teuthology.orchestra.run.smithi111.stdout:Ign:2 https://4.chacra.ceph.com/r/ceph/wip-ajarr-63009-only-reacquire-kickstart/a7de91bfa5778a0d559a7e9bc8e35f93456b41ad/ubuntu/jammy/flavors/default jammy InRelease 2023-10-03T17:34:51.031 INFO:teuthology.orchestra.run.smithi111.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2023-10-03T17:34:51.051 INFO:teuthology.orchestra.run.smithi111.stdout:Hit:4 https://4.chacra.ceph.com/r/ceph/wip-ajarr-63009-only-reacquire-kickstart/a7de91bfa5778a0d559a7e9bc8e35f93456b41ad/ubuntu/jammy/flavors/default jammy Release 2023-10-03T17:34:51.063 INFO:teuthology.orchestra.run.smithi111.stdout:Hit:5 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2023-10-03T17:34:51.072 INFO:teuthology.orchestra.run.smithi111.stdout:Ign:6 https://4.chacra.ceph.com/r/ceph/wip-ajarr-63009-only-reacquire-kickstart/a7de91bfa5778a0d559a7e9bc8e35f93456b41ad/ubuntu/jammy/flavors/default jammy Release.gpg 2023-10-03T17:34:57.907 INFO:teuthology.orchestra.run.smithi111.stdout:Hit:7 http://security.ubuntu.com/ubuntu jammy-security InRelease 2023-10-03T17:34:58.976 INFO:teuthology.orchestra.run.smithi111.stdout:Reading package lists... 2023-10-03T17:34:58.989 DEBUG:teuthology.orchestra.run.smithi111:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=18.0.0-6336-ga7de91bf-1jammy cephadm=18.0.0-6336-ga7de91bf-1jammy ceph-mds=18.0.0-6336-ga7de91bf-1jammy ceph-mgr=18.0.0-6336-ga7de91bf-1jammy ceph-common=18.0.0-6336-ga7de91bf-1jammy ceph-fuse=18.0.0-6336-ga7de91bf-1jammy ceph-test=18.0.0-6336-ga7de91bf-1jammy ceph-volume=18.0.0-6336-ga7de91bf-1jammy radosgw=18.0.0-6336-ga7de91bf-1jammy python3-rados=18.0.0-6336-ga7de91bf-1jammy python3-rgw=18.0.0-6336-ga7de91bf-1jammy python3-cephfs=18.0.0-6336-ga7de91bf-1jammy python3-rbd=18.0.0-6336-ga7de91bf-1jammy libcephfs2=18.0.0-6336-ga7de91bf-1jammy libcephfs-dev=18.0.0-6336-ga7de91bf-1jammy librados2=18.0.0-6336-ga7de91bf-1jammy librbd1=18.0.0-6336-ga7de91bf-1jammy rbd-fuse=18.0.0-6336-ga7de91bf-1jammy 2023-10-03T17:34:59.042 INFO:teuthology.orchestra.run.smithi111.stdout:Reading package lists... 2023-10-03T17:34:59.208 INFO:teuthology.orchestra.run.smithi111.stdout:Building dependency tree... 2023-10-03T17:34:59.208 INFO:teuthology.orchestra.run.smithi111.stdout:Reading state information... 2023-10-03T17:34:59.351 INFO:teuthology.orchestra.run.smithi111.stdout:ceph is already the newest version (18.0.0-6336-ga7de91bf-1jammy). 2023-10-03T17:34:59.352 INFO:teuthology.orchestra.run.smithi111.stdout:ceph-common is already the newest version (18.0.0-6336-ga7de91bf-1jammy). 2023-10-03T17:34:59.352 INFO:teuthology.orchestra.run.smithi111.stdout:ceph-fuse is already the newest version (18.0.0-6336-ga7de91bf-1jammy). 2023-10-03T17:34:59.352 INFO:teuthology.orchestra.run.smithi111.stdout:ceph-mds is already the newest version (18.0.0-6336-ga7de91bf-1jammy). 2023-10-03T17:34:59.352 INFO:teuthology.orchestra.run.smithi111.stdout:ceph-mgr is already the newest version (18.0.0-6336-ga7de91bf-1jammy). 2023-10-03T17:34:59.352 INFO:teuthology.orchestra.run.smithi111.stdout:ceph-test is already the newest version (18.0.0-6336-ga7de91bf-1jammy). 2023-10-03T17:34:59.352 INFO:teuthology.orchestra.run.smithi111.stdout:ceph-volume is already the newest version (18.0.0-6336-ga7de91bf-1jammy). 2023-10-03T17:34:59.352 INFO:teuthology.orchestra.run.smithi111.stdout:cephadm is already the newest version (18.0.0-6336-ga7de91bf-1jammy). 2023-10-03T17:34:59.352 INFO:teuthology.orchestra.run.smithi111.stdout:libcephfs-dev is already the newest version (18.0.0-6336-ga7de91bf-1jammy). 2023-10-03T17:34:59.352 INFO:teuthology.orchestra.run.smithi111.stdout:libcephfs2 is already the newest version (18.0.0-6336-ga7de91bf-1jammy). 2023-10-03T17:34:59.353 INFO:teuthology.orchestra.run.smithi111.stdout:librados2 is already the newest version (18.0.0-6336-ga7de91bf-1jammy). 2023-10-03T17:34:59.353 INFO:teuthology.orchestra.run.smithi111.stdout:librbd1 is already the newest version (18.0.0-6336-ga7de91bf-1jammy). 2023-10-03T17:34:59.353 INFO:teuthology.orchestra.run.smithi111.stdout:python3-cephfs is already the newest version (18.0.0-6336-ga7de91bf-1jammy). 2023-10-03T17:34:59.353 INFO:teuthology.orchestra.run.smithi111.stdout:python3-rados is already the newest version (18.0.0-6336-ga7de91bf-1jammy). 2023-10-03T17:34:59.353 INFO:teuthology.orchestra.run.smithi111.stdout:python3-rbd is already the newest version (18.0.0-6336-ga7de91bf-1jammy). 2023-10-03T17:34:59.353 INFO:teuthology.orchestra.run.smithi111.stdout:python3-rgw is already the newest version (18.0.0-6336-ga7de91bf-1jammy). 2023-10-03T17:34:59.353 INFO:teuthology.orchestra.run.smithi111.stdout:radosgw is already the newest version (18.0.0-6336-ga7de91bf-1jammy). 2023-10-03T17:34:59.353 INFO:teuthology.orchestra.run.smithi111.stdout:rbd-fuse is already the newest version (18.0.0-6336-ga7de91bf-1jammy). 2023-10-03T17:34:59.353 INFO:teuthology.orchestra.run.smithi111.stdout:The following packages were automatically installed and are no longer required: 2023-10-03T17:34:59.353 INFO:teuthology.orchestra.run.smithi111.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-10-03T17:34:59.353 INFO:teuthology.orchestra.run.smithi111.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-10-03T17:34:59.354 INFO:teuthology.orchestra.run.smithi111.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-10-03T17:34:59.354 INFO:teuthology.orchestra.run.smithi111.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2023-10-03T17:34:59.354 INFO:teuthology.orchestra.run.smithi111.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2023-10-03T17:34:59.354 INFO:teuthology.orchestra.run.smithi111.stdout:Use 'sudo apt autoremove' to remove them. 2023-10-03T17:34:59.387 INFO:teuthology.orchestra.run.smithi111.stdout:0 upgraded, 0 newly installed, 0 to remove and 254 not upgraded. 2023-10-03T17:34:59.387 INFO:teuthology.orchestra.run.smithi111.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-10-03T17:34:59.389 DEBUG:teuthology.orchestra.run.smithi111:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install fio 2023-10-03T17:34:59.494 INFO:teuthology.orchestra.run.smithi111.stdout:Reading package lists... 2023-10-03T17:34:59.679 INFO:teuthology.orchestra.run.smithi111.stdout:Building dependency tree... 2023-10-03T17:34:59.680 INFO:teuthology.orchestra.run.smithi111.stdout:Reading state information... 2023-10-03T17:34:59.838 INFO:teuthology.orchestra.run.smithi111.stdout:The following packages were automatically installed and are no longer required: 2023-10-03T17:34:59.838 INFO:teuthology.orchestra.run.smithi111.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libiscsi7 libpmemobj1 2023-10-03T17:34:59.838 INFO:teuthology.orchestra.run.smithi111.stdout: libpython2-dev libpython2-stdlib libpython2.7 libpython2.7-dev 2023-10-03T17:34:59.838 INFO:teuthology.orchestra.run.smithi111.stdout: libpython2.7-minimal libpython2.7-stdlib linux-image-5.4.0-135-generic 2023-10-03T17:34:59.839 INFO:teuthology.orchestra.run.smithi111.stdout: linux-modules-5.4.0-135-generic python2 python2-dev python2-minimal 2023-10-03T17:34:59.839 INFO:teuthology.orchestra.run.smithi111.stdout: python2.7 python2.7-dev python2.7-minimal 2023-10-03T17:34:59.839 INFO:teuthology.orchestra.run.smithi111.stdout:Use 'sudo apt autoremove' to remove them. 2023-10-03T17:34:59.839 INFO:teuthology.orchestra.run.smithi111.stdout:The following additional packages will be installed: 2023-10-03T17:34:59.840 INFO:teuthology.orchestra.run.smithi111.stdout: libpmemblk1 2023-10-03T17:34:59.841 INFO:teuthology.orchestra.run.smithi111.stdout:Suggested packages: 2023-10-03T17:34:59.841 INFO:teuthology.orchestra.run.smithi111.stdout: gnuplot gfio python-scipy 2023-10-03T17:34:59.867 INFO:teuthology.orchestra.run.smithi111.stdout:The following NEW packages will be installed: 2023-10-03T17:34:59.868 INFO:teuthology.orchestra.run.smithi111.stdout: fio libpmemblk1 2023-10-03T17:35:00.047 INFO:teuthology.orchestra.run.smithi111.stdout:0 upgraded, 2 newly installed, 0 to remove and 254 not upgraded. 2023-10-03T17:35:00.048 INFO:teuthology.orchestra.run.smithi111.stdout:Need to get 4,112 kB of archives. 2023-10-03T17:35:00.048 INFO:teuthology.orchestra.run.smithi111.stdout:After this operation, 7,813 kB of additional disk space will be used. 2023-10-03T17:35:00.048 INFO:teuthology.orchestra.run.smithi111.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libpmemblk1 amd64 1.11.1-3build1 [65.6 kB] 2023-10-03T17:35:00.331 INFO:teuthology.orchestra.run.smithi111.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/universe amd64 fio amd64 3.28-1 [4,047 kB] 2023-10-03T17:35:01.203 INFO:teuthology.orchestra.run.smithi111.stdout:Fetched 4,112 kB in 1s (4,277 kB/s) 2023-10-03T17:35:01.362 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package libpmemblk1:amd64. 2023-10-03T17:35:01.459 INFO:teuthology.orchestra.run.smithi111.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 172896 files and directories currently installed.) 2023-10-03T17:35:01.465 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../libpmemblk1_1.11.1-3build1_amd64.deb ... 2023-10-03T17:35:01.552 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking libpmemblk1:amd64 (1.11.1-3build1) ... 2023-10-03T17:35:01.897 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package fio. 2023-10-03T17:35:01.925 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../archives/fio_3.28-1_amd64.deb ... 2023-10-03T17:35:01.964 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking fio (3.28-1) ... 2023-10-03T17:35:02.474 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up libpmemblk1:amd64 (1.11.1-3build1) ... 2023-10-03T17:35:02.595 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up fio (3.28-1) ... 2023-10-03T17:35:03.156 INFO:teuthology.orchestra.run.smithi111.stdout:Processing triggers for man-db (2.10.2-1) ... 2023-10-03T17:35:04.165 INFO:teuthology.orchestra.run.smithi111.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2023-10-03T17:35:04.648 INFO:teuthology.orchestra.run.smithi111.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-10-03T17:35:04.651 DEBUG:teuthology.parallel:result is None 2023-10-03T17:35:04.651 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=a7de91bfa5778a0d559a7e9bc8e35f93456b41ad 2023-10-03T17:35:04.799 DEBUG:teuthology.orchestra.run.smithi111:> dpkg-query -W -f '${Version}' ceph 2023-10-03T17:35:04.818 INFO:teuthology.orchestra.run.smithi111.stdout:18.0.0-6336-ga7de91bf-1jammy 2023-10-03T17:35:04.818 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-6336-ga7de91bf-1jammy 2023-10-03T17:35:04.819 INFO:teuthology.task.install:The correct ceph version 18.0.0-6336-ga7de91bf-1jammy is installed. 2023-10-03T17:35:04.819 INFO:teuthology.task.install.util:Utilities already shipped, skip it... 2023-10-03T17:35:04.819 INFO:teuthology.run_tasks:Running task workunit... 2023-10-03T17:35:04.831 INFO:tasks.workunit:Pulling workunits from ref a7de91bfa5778a0d559a7e9bc8e35f93456b41ad 2023-10-03T17:35:04.832 INFO:tasks.workunit:Making a separate scratch dir for every client... 2023-10-03T17:35:04.832 DEBUG:teuthology.orchestra.run.smithi111:> stat -- /home/ubuntu/cephtest/mnt.0 2023-10-03T17:35:04.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-03T17:35:04.867 INFO:teuthology.orchestra.run.smithi111.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2023-10-03T17:35:04.868 DEBUG:teuthology.orchestra.run.smithi111:> mkdir -- /home/ubuntu/cephtest/mnt.0 2023-10-03T17:35:04.915 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2023-10-03T17:35:04.915 DEBUG:teuthology.orchestra.run.smithi111:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2023-10-03T17:35:04.963 INFO:tasks.workunit:timeout=3h 2023-10-03T17:35:04.963 INFO:tasks.workunit:cleanup=True 2023-10-03T17:35:04.964 DEBUG:teuthology.orchestra.run.smithi111:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout a7de91bfa5778a0d559a7e9bc8e35f93456b41ad 2023-10-03T17:35:05.013 INFO:tasks.workunit.client.0.smithi111.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2023-10-03T17:36:02.744 INFO:tasks.workunit.client.0.smithi111.stderr:Note: switching to 'a7de91bfa5778a0d559a7e9bc8e35f93456b41ad'. 2023-10-03T17:36:02.744 INFO:tasks.workunit.client.0.smithi111.stderr: 2023-10-03T17:36:02.745 INFO:tasks.workunit.client.0.smithi111.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2023-10-03T17:36:02.745 INFO:tasks.workunit.client.0.smithi111.stderr:changes and commit them, and you can discard any commits you make in this 2023-10-03T17:36:02.745 INFO:tasks.workunit.client.0.smithi111.stderr:state without impacting any branches by switching back to a branch. 2023-10-03T17:36:02.745 INFO:tasks.workunit.client.0.smithi111.stderr: 2023-10-03T17:36:02.745 INFO:tasks.workunit.client.0.smithi111.stderr:If you want to create a new branch to retain commits you create, you may 2023-10-03T17:36:02.745 INFO:tasks.workunit.client.0.smithi111.stderr:do so (now or later) by using -c with the switch command. Example: 2023-10-03T17:36:02.745 INFO:tasks.workunit.client.0.smithi111.stderr: 2023-10-03T17:36:02.745 INFO:tasks.workunit.client.0.smithi111.stderr: git switch -c 2023-10-03T17:36:02.745 INFO:tasks.workunit.client.0.smithi111.stderr: 2023-10-03T17:36:02.745 INFO:tasks.workunit.client.0.smithi111.stderr:Or undo this operation with: 2023-10-03T17:36:02.745 INFO:tasks.workunit.client.0.smithi111.stderr: 2023-10-03T17:36:02.745 INFO:tasks.workunit.client.0.smithi111.stderr: git switch - 2023-10-03T17:36:02.746 INFO:tasks.workunit.client.0.smithi111.stderr: 2023-10-03T17:36:02.746 INFO:tasks.workunit.client.0.smithi111.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2023-10-03T17:36:02.746 INFO:tasks.workunit.client.0.smithi111.stderr: 2023-10-03T17:36:02.746 INFO:tasks.workunit.client.0.smithi111.stderr:HEAD is now at a7de91bfa57 librbd: fix another block when waiting for exclusive lock 2023-10-03T17:36:02.747 DEBUG:teuthology.orchestra.run.smithi111:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2023-10-03T17:36:02.794 INFO:tasks.workunit.client.0.smithi111.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2023-10-03T17:36:02.798 INFO:tasks.workunit.client.0.smithi111.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-10-03T17:36:02.798 INFO:tasks.workunit.client.0.smithi111.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2023-10-03T17:36:03.514 INFO:tasks.workunit.client.0.smithi111.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2023-10-03T17:36:03.596 INFO:tasks.workunit.client.0.smithi111.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2023-10-03T17:36:03.646 INFO:tasks.workunit.client.0.smithi111.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-10-03T17:36:03.647 INFO:tasks.workunit.client.0.smithi111.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-10-03T17:36:03.647 INFO:tasks.workunit.client.0.smithi111.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2023-10-03T17:36:03.698 INFO:tasks.workunit.client.0.smithi111.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-10-03T17:36:03.701 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-10-03T17:36:03.701 DEBUG:teuthology.orchestra.run.smithi111:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2023-10-03T17:36:03.750 INFO:tasks.workunit:Running workunits matching rbd/rbd_support_module_recovery.sh on client.0... 2023-10-03T17:36:03.752 INFO:tasks.workunit:Running workunit rbd/rbd_support_module_recovery.sh... 2023-10-03T17:36:03.752 DEBUG:teuthology.orchestra.run.smithi111:workunit test rbd/rbd_support_module_recovery.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a7de91bfa5778a0d559a7e9bc8e35f93456b41ad TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/rbd_support_module_recovery.sh 2023-10-03T17:36:03.802 INFO:tasks.workunit.client.0.smithi111.stderr:+ POOL=rbd 2023-10-03T17:36:03.803 INFO:tasks.workunit.client.0.smithi111.stderr:+ IMAGE_PREFIX=image 2023-10-03T17:36:03.803 INFO:tasks.workunit.client.0.smithi111.stderr:+ NUM_IMAGES=20 2023-10-03T17:36:03.803 INFO:tasks.workunit.client.0.smithi111.stderr:+ RUN_TIME=3600 2023-10-03T17:36:03.803 INFO:tasks.workunit.client.0.smithi111.stderr:+ rbd mirror pool enable rbd image 2023-10-03T17:36:03.851 INFO:tasks.workunit.client.0.smithi111.stderr:+ rbd mirror pool peer add rbd dummy 2023-10-03T17:36:03.888 INFO:tasks.workunit.client.0.smithi111.stdout:118bc2d9-6e11-4d73-91f1-839785745a5f 2023-10-03T17:36:03.892 INFO:tasks.workunit.client.0.smithi111.stderr:+ (( i = 1 )) 2023-10-03T17:36:03.892 INFO:tasks.workunit.client.0.smithi111.stderr:+ (( i <= 20 )) 2023-10-03T17:36:03.892 INFO:tasks.workunit.client.0.smithi111.stderr:+ rbd create --size 1G rbd/image1 2023-10-03T17:36:03.937 INFO:tasks.workunit.client.0.smithi111.stderr:+ rbd mirror image enable rbd/image1 snapshot 2023-10-03T17:36:04.245 INFO:tasks.workunit.client.0.smithi111.stdout:Mirroring enabled 2023-10-03T17:36:04.254 INFO:tasks.workunit.client.0.smithi111.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 1m 2023-10-03T17:36:04.316 INFO:tasks.workunit.client.0.smithi111.stderr:+ (( i++ )) 2023-10-03T17:36:04.317 INFO:tasks.workunit.client.0.smithi111.stderr:+ (( i <= 20 )) 2023-10-03T17:36:04.317 INFO:tasks.workunit.client.0.smithi111.stderr:+ rbd create --size 1G rbd/image2 2023-10-03T17:36:04.363 INFO:tasks.workunit.client.0.smithi111.stderr:+ rbd mirror image enable rbd/image2 snapshot 2023-10-03T17:36:05.241 INFO:tasks.workunit.client.0.smithi111.stdout:Mirroring enabled 2023-10-03T17:36:05.251 INFO:tasks.workunit.client.0.smithi111.stderr:+ rbd mirror snapshot schedule add -p rbd --image image2 1m 2023-10-03T17:36:05.311 INFO:tasks.workunit.client.0.smithi111.stderr:+ (( i++ )) 2023-10-03T17:36:05.311 INFO:tasks.workunit.client.0.smithi111.stderr:+ (( i <= 20 )) 2023-10-03T17:36:05.311 INFO:tasks.workunit.client.0.smithi111.stderr:+ rbd create --size 1G rbd/image3 2023-10-03T17:36:05.356 INFO:tasks.workunit.client.0.smithi111.stderr:+ rbd mirror image enable rbd/image3 snapshot 2023-10-03T17:36:06.253 INFO:tasks.workunit.client.0.smithi111.stdout:Mirroring enabled 2023-10-03T17:36:06.264 INFO:tasks.workunit.client.0.smithi111.stderr:+ rbd mirror snapshot schedule add -p rbd --image image3 1m 2023-10-03T17:36:06.327 INFO:tasks.workunit.client.0.smithi111.stderr:+ (( i++ )) 2023-10-03T17:36:06.327 INFO:tasks.workunit.client.0.smithi111.stderr:+ (( i <= 20 )) 2023-10-03T17:36:06.327 INFO:tasks.workunit.client.0.smithi111.stderr:+ rbd create --size 1G rbd/image4 2023-10-03T17:36:06.380 INFO:tasks.workunit.client.0.smithi111.stderr:+ rbd mirror image enable rbd/image4 snapshot 2023-10-03T17:36:07.253 INFO:tasks.workunit.client.0.smithi111.stdout:Mirroring enabled 2023-10-03T17:36:07.263 INFO:tasks.workunit.client.0.smithi111.stderr:+ rbd mirror snapshot schedule add -p rbd --image image4 1m 2023-10-03T17:36:07.325 INFO:tasks.workunit.client.0.smithi111.stderr:+ (( i++ )) 2023-10-03T17:36:07.325 INFO:tasks.workunit.client.0.smithi111.stderr:+ (( i <= 20 )) 2023-10-03T17:36:07.325 INFO:tasks.workunit.client.0.smithi111.stderr:+ rbd create --size 1G rbd/image5 2023-10-03T17:36:07.375 INFO:tasks.workunit.client.0.smithi111.stderr:+ rbd mirror image enable rbd/image5 snapshot 2023-10-03T17:36:08.270 INFO:tasks.workunit.client.0.smithi111.stdout:Mirroring enabled 2023-10-03T17:36:08.282 INFO:tasks.workunit.client.0.smithi111.stderr:+ rbd mirror snapshot schedule add -p rbd --image image5 1m 2023-10-03T17:36:08.349 INFO:tasks.workunit.client.0.smithi111.stderr:+ (( i++ )) 2023-10-03T17:36:08.349 INFO:tasks.workunit.client.0.smithi111.stderr:+ (( i <= 20 )) 2023-10-03T17:36:08.349 INFO:tasks.workunit.client.0.smithi111.stderr:+ rbd create --size 1G rbd/image6 2023-10-03T17:36:08.397 INFO:tasks.workunit.client.0.smithi111.stderr:+ rbd mirror image enable rbd/image6 snapshot 2023-10-03T17:36:09.278 INFO:tasks.workunit.client.0.smithi111.stdout:Mirroring enabled 2023-10-03T17:36:09.290 INFO:tasks.workunit.client.0.smithi111.stderr:+ rbd mirror snapshot schedule add -p rbd --image image6 1m 2023-10-03T17:36:09.351 INFO:tasks.workunit.client.0.smithi111.stderr:+ (( i++ )) 2023-10-03T17:36:09.351 INFO:tasks.workunit.client.0.smithi111.stderr:+ (( i <= 20 )) 2023-10-03T17:36:09.352 INFO:tasks.workunit.client.0.smithi111.stderr:+ rbd create --size 1G rbd/image7 2023-10-03T17:36:09.401 INFO:tasks.workunit.client.0.smithi111.stderr:+ rbd mirror image enable rbd/image7 snapshot 2023-10-03T17:36:10.287 INFO:tasks.workunit.client.0.smithi111.stdout:Mirroring enabled 2023-10-03T17:36:10.297 INFO:tasks.workunit.client.0.smithi111.stderr:+ rbd mirror snapshot schedule add -p rbd --image image7 1m 2023-10-03T17:36:10.355 INFO:tasks.workunit.client.0.smithi111.stderr:+ (( i++ )) 2023-10-03T17:36:10.356 INFO:tasks.workunit.client.0.smithi111.stderr:+ (( i <= 20 )) 2023-10-03T17:36:10.356 INFO:tasks.workunit.client.0.smithi111.stderr:+ rbd create --size 1G rbd/image8 2023-10-03T17:36:10.402 INFO:tasks.workunit.client.0.smithi111.stderr:+ rbd mirror image enable rbd/image8 snapshot 2023-10-03T17:36:11.292 INFO:tasks.workunit.client.0.smithi111.stdout:Mirroring enabled 2023-10-03T17:36:11.301 INFO:tasks.workunit.client.0.smithi111.stderr:+ rbd mirror snapshot schedule add -p rbd --image image8 1m 2023-10-03T17:36:11.362 INFO:tasks.workunit.client.0.smithi111.stderr:+ (( i++ )) 2023-10-03T17:36:11.362 INFO:tasks.workunit.client.0.smithi111.stderr:+ (( i <= 20 )) 2023-10-03T17:36:11.362 INFO:tasks.workunit.client.0.smithi111.stderr:+ rbd create --size 1G rbd/image9 2023-10-03T17:36:11.408 INFO:tasks.workunit.client.0.smithi111.stderr:+ rbd mirror image enable rbd/image9 snapshot 2023-10-03T17:36:12.305 INFO:tasks.workunit.client.0.smithi111.stdout:Mirroring enabled 2023-10-03T17:36:12.315 INFO:tasks.workunit.client.0.smithi111.stderr:+ rbd mirror snapshot schedule add -p rbd --image image9 1m 2023-10-03T17:36:12.377 INFO:tasks.workunit.client.0.smithi111.stderr:+ (( i++ )) 2023-10-03T17:36:12.377 INFO:tasks.workunit.client.0.smithi111.stderr:+ (( i <= 20 )) 2023-10-03T17:36:12.378 INFO:tasks.workunit.client.0.smithi111.stderr:+ rbd create --size 1G rbd/image10 2023-10-03T17:36:12.426 INFO:tasks.workunit.client.0.smithi111.stderr:+ rbd mirror image enable rbd/image10 snapshot 2023-10-03T17:36:13.317 INFO:tasks.workunit.client.0.smithi111.stdout:Mirroring enabled 2023-10-03T17:36:13.327 INFO:tasks.workunit.client.0.smithi111.stderr:+ rbd mirror snapshot schedule add -p rbd --image image10 1m 2023-10-03T17:36:13.391 INFO:tasks.workunit.client.0.smithi111.stderr:+ (( i++ )) 2023-10-03T17:36:13.391 INFO:tasks.workunit.client.0.smithi111.stderr:+ (( i <= 20 )) 2023-10-03T17:36:13.391 INFO:tasks.workunit.client.0.smithi111.stderr:+ rbd create --size 1G rbd/image11 2023-10-03T17:36:13.438 INFO:tasks.workunit.client.0.smithi111.stderr:+ rbd mirror image enable rbd/image11 snapshot 2023-10-03T17:36:14.316 INFO:tasks.workunit.client.0.smithi111.stdout:Mirroring enabled 2023-10-03T17:36:14.324 INFO:tasks.workunit.client.0.smithi111.stderr:+ rbd mirror snapshot schedule add -p rbd --image image11 1m 2023-10-03T17:36:14.383 INFO:tasks.workunit.client.0.smithi111.stderr:+ (( i++ )) 2023-10-03T17:36:14.384 INFO:tasks.workunit.client.0.smithi111.stderr:+ (( i <= 20 )) 2023-10-03T17:36:14.384 INFO:tasks.workunit.client.0.smithi111.stderr:+ rbd create --size 1G rbd/image12 2023-10-03T17:36:14.431 INFO:tasks.workunit.client.0.smithi111.stderr:+ rbd mirror image enable rbd/image12 snapshot 2023-10-03T17:36:15.320 INFO:tasks.workunit.client.0.smithi111.stdout:Mirroring enabled 2023-10-03T17:36:15.329 INFO:tasks.workunit.client.0.smithi111.stderr:+ rbd mirror snapshot schedule add -p rbd --image image12 1m 2023-10-03T17:36:15.390 INFO:tasks.workunit.client.0.smithi111.stderr:+ (( i++ )) 2023-10-03T17:36:15.391 INFO:tasks.workunit.client.0.smithi111.stderr:+ (( i <= 20 )) 2023-10-03T17:36:15.391 INFO:tasks.workunit.client.0.smithi111.stderr:+ rbd create --size 1G rbd/image13 2023-10-03T17:36:15.437 INFO:tasks.workunit.client.0.smithi111.stderr:+ rbd mirror image enable rbd/image13 snapshot 2023-10-03T17:36:16.333 INFO:tasks.workunit.client.0.smithi111.stdout:Mirroring enabled 2023-10-03T17:36:16.342 INFO:tasks.workunit.client.0.smithi111.stderr:+ rbd mirror snapshot schedule add -p rbd --image image13 1m 2023-10-03T17:36:16.404 INFO:tasks.workunit.client.0.smithi111.stderr:+ (( i++ )) 2023-10-03T17:36:16.404 INFO:tasks.workunit.client.0.smithi111.stderr:+ (( i <= 20 )) 2023-10-03T17:36:16.404 INFO:tasks.workunit.client.0.smithi111.stderr:+ rbd create --size 1G rbd/image14 2023-10-03T17:36:16.454 INFO:tasks.workunit.client.0.smithi111.stderr:+ rbd mirror image enable rbd/image14 snapshot 2023-10-03T17:36:17.338 INFO:tasks.workunit.client.0.smithi111.stdout:Mirroring enabled 2023-10-03T17:36:17.349 INFO:tasks.workunit.client.0.smithi111.stderr:+ rbd mirror snapshot schedule add -p rbd --image image14 1m 2023-10-03T17:36:17.410 INFO:tasks.workunit.client.0.smithi111.stderr:+ (( i++ )) 2023-10-03T17:36:17.410 INFO:tasks.workunit.client.0.smithi111.stderr:+ (( i <= 20 )) 2023-10-03T17:36:17.410 INFO:tasks.workunit.client.0.smithi111.stderr:+ rbd create --size 1G rbd/image15 2023-10-03T17:36:17.456 INFO:tasks.workunit.client.0.smithi111.stderr:+ rbd mirror image enable rbd/image15 snapshot 2023-10-03T17:36:18.353 INFO:tasks.workunit.client.0.smithi111.stdout:Mirroring enabled 2023-10-03T17:36:18.360 INFO:tasks.workunit.client.0.smithi111.stderr:+ rbd mirror snapshot schedule add -p rbd --image image15 1m 2023-10-03T17:36:18.421 INFO:tasks.workunit.client.0.smithi111.stderr:+ (( i++ )) 2023-10-03T17:36:18.421 INFO:tasks.workunit.client.0.smithi111.stderr:+ (( i <= 20 )) 2023-10-03T17:36:18.421 INFO:tasks.workunit.client.0.smithi111.stderr:+ rbd create --size 1G rbd/image16 2023-10-03T17:36:18.468 INFO:tasks.workunit.client.0.smithi111.stderr:+ rbd mirror image enable rbd/image16 snapshot 2023-10-03T17:36:19.363 INFO:tasks.workunit.client.0.smithi111.stdout:Mirroring enabled 2023-10-03T17:36:19.373 INFO:tasks.workunit.client.0.smithi111.stderr:+ rbd mirror snapshot schedule add -p rbd --image image16 1m 2023-10-03T17:36:19.434 INFO:tasks.workunit.client.0.smithi111.stderr:+ (( i++ )) 2023-10-03T17:36:19.434 INFO:tasks.workunit.client.0.smithi111.stderr:+ (( i <= 20 )) 2023-10-03T17:36:19.435 INFO:tasks.workunit.client.0.smithi111.stderr:+ rbd create --size 1G rbd/image17 2023-10-03T17:36:19.481 INFO:tasks.workunit.client.0.smithi111.stderr:+ rbd mirror image enable rbd/image17 snapshot 2023-10-03T17:36:20.366 INFO:tasks.workunit.client.0.smithi111.stdout:Mirroring enabled 2023-10-03T17:36:20.376 INFO:tasks.workunit.client.0.smithi111.stderr:+ rbd mirror snapshot schedule add -p rbd --image image17 1m 2023-10-03T17:36:20.438 INFO:tasks.workunit.client.0.smithi111.stderr:+ (( i++ )) 2023-10-03T17:36:20.438 INFO:tasks.workunit.client.0.smithi111.stderr:+ (( i <= 20 )) 2023-10-03T17:36:20.438 INFO:tasks.workunit.client.0.smithi111.stderr:+ rbd create --size 1G rbd/image18 2023-10-03T17:36:20.488 INFO:tasks.workunit.client.0.smithi111.stderr:+ rbd mirror image enable rbd/image18 snapshot 2023-10-03T17:36:21.376 INFO:tasks.workunit.client.0.smithi111.stdout:Mirroring enabled 2023-10-03T17:36:21.384 INFO:tasks.workunit.client.0.smithi111.stderr:+ rbd mirror snapshot schedule add -p rbd --image image18 1m 2023-10-03T17:36:21.445 INFO:tasks.workunit.client.0.smithi111.stderr:+ (( i++ )) 2023-10-03T17:36:21.445 INFO:tasks.workunit.client.0.smithi111.stderr:+ (( i <= 20 )) 2023-10-03T17:36:21.446 INFO:tasks.workunit.client.0.smithi111.stderr:+ rbd create --size 1G rbd/image19 2023-10-03T17:36:21.494 INFO:tasks.workunit.client.0.smithi111.stderr:+ rbd mirror image enable rbd/image19 snapshot 2023-10-03T17:36:22.381 INFO:tasks.workunit.client.0.smithi111.stdout:Mirroring enabled 2023-10-03T17:36:22.394 INFO:tasks.workunit.client.0.smithi111.stderr:+ rbd mirror snapshot schedule add -p rbd --image image19 1m 2023-10-03T17:36:22.456 INFO:tasks.workunit.client.0.smithi111.stderr:+ (( i++ )) 2023-10-03T17:36:22.456 INFO:tasks.workunit.client.0.smithi111.stderr:+ (( i <= 20 )) 2023-10-03T17:36:22.456 INFO:tasks.workunit.client.0.smithi111.stderr:+ rbd create --size 1G rbd/image20 2023-10-03T17:36:22.506 INFO:tasks.workunit.client.0.smithi111.stderr:+ rbd mirror image enable rbd/image20 snapshot 2023-10-03T17:36:23.393 INFO:tasks.workunit.client.0.smithi111.stdout:Mirroring enabled 2023-10-03T17:36:23.403 INFO:tasks.workunit.client.0.smithi111.stderr:+ rbd mirror snapshot schedule add -p rbd --image image20 1m 2023-10-03T17:36:23.468 INFO:tasks.workunit.client.0.smithi111.stderr:+ (( i++ )) 2023-10-03T17:36:23.468 INFO:tasks.workunit.client.0.smithi111.stderr:+ (( i <= 20 )) 2023-10-03T17:36:23.468 INFO:tasks.workunit.client.0.smithi111.stderr:+ (( i = 1 )) 2023-10-03T17:36:23.468 INFO:tasks.workunit.client.0.smithi111.stderr:+ (( i <= 20 )) 2023-10-03T17:36:23.468 INFO:tasks.workunit.client.0.smithi111.stderr:++ sudo rbd device map rbd/image1 2023-10-03T17:36:23.594 INFO:tasks.workunit.client.0.smithi111.stderr:+ DEVS[$i]=/dev/rbd0 2023-10-03T17:36:23.594 INFO:tasks.workunit.client.0.smithi111.stderr:+ (( i++ )) 2023-10-03T17:36:23.594 INFO:tasks.workunit.client.0.smithi111.stderr:+ (( i <= 20 )) 2023-10-03T17:36:23.594 INFO:tasks.workunit.client.0.smithi111.stderr:+ fio --name=fiotest --filename=/dev/rbd0 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-03T17:36:23.594 INFO:tasks.workunit.client.0.smithi111.stderr:++ sudo rbd device map rbd/image2 2023-10-03T17:36:23.664 INFO:tasks.workunit.client.0.smithi111.stderr:+ DEVS[$i]=/dev/rbd1 2023-10-03T17:36:23.664 INFO:tasks.workunit.client.0.smithi111.stderr:+ (( i++ )) 2023-10-03T17:36:23.664 INFO:tasks.workunit.client.0.smithi111.stderr:+ (( i <= 20 )) 2023-10-03T17:36:23.664 INFO:tasks.workunit.client.0.smithi111.stderr:+ fio --name=fiotest --filename=/dev/rbd1 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-03T17:36:23.664 INFO:tasks.workunit.client.0.smithi111.stderr:++ sudo rbd device map rbd/image3 2023-10-03T17:36:23.735 INFO:tasks.workunit.client.0.smithi111.stderr:+ DEVS[$i]=/dev/rbd2 2023-10-03T17:36:23.735 INFO:tasks.workunit.client.0.smithi111.stderr:+ (( i++ )) 2023-10-03T17:36:23.735 INFO:tasks.workunit.client.0.smithi111.stderr:+ (( i <= 20 )) 2023-10-03T17:36:23.735 INFO:tasks.workunit.client.0.smithi111.stderr:+ fio --name=fiotest --filename=/dev/rbd2 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-03T17:36:23.735 INFO:tasks.workunit.client.0.smithi111.stderr:++ sudo rbd device map rbd/image4 2023-10-03T17:36:23.810 INFO:tasks.workunit.client.0.smithi111.stderr:+ DEVS[$i]=/dev/rbd3 2023-10-03T17:36:23.810 INFO:tasks.workunit.client.0.smithi111.stderr:+ (( i++ )) 2023-10-03T17:36:23.810 INFO:tasks.workunit.client.0.smithi111.stderr:+ (( i <= 20 )) 2023-10-03T17:36:23.810 INFO:tasks.workunit.client.0.smithi111.stderr:+ fio --name=fiotest --filename=/dev/rbd3 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-03T17:36:23.810 INFO:tasks.workunit.client.0.smithi111.stderr:++ sudo rbd device map rbd/image5 2023-10-03T17:36:23.880 INFO:tasks.workunit.client.0.smithi111.stderr:+ DEVS[$i]=/dev/rbd4 2023-10-03T17:36:23.881 INFO:tasks.workunit.client.0.smithi111.stderr:+ (( i++ )) 2023-10-03T17:36:23.881 INFO:tasks.workunit.client.0.smithi111.stderr:+ (( i <= 20 )) 2023-10-03T17:36:23.881 INFO:tasks.workunit.client.0.smithi111.stderr:+ fio --name=fiotest --filename=/dev/rbd4 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-03T17:36:23.881 INFO:tasks.workunit.client.0.smithi111.stderr:++ sudo rbd device map rbd/image6 2023-10-03T17:36:23.969 INFO:tasks.workunit.client.0.smithi111.stderr:+ DEVS[$i]=/dev/rbd5 2023-10-03T17:36:23.969 INFO:tasks.workunit.client.0.smithi111.stderr:+ (( i++ )) 2023-10-03T17:36:23.969 INFO:tasks.workunit.client.0.smithi111.stderr:+ (( i <= 20 )) 2023-10-03T17:36:23.969 INFO:tasks.workunit.client.0.smithi111.stderr:+ fio --name=fiotest --filename=/dev/rbd5 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-03T17:36:23.969 INFO:tasks.workunit.client.0.smithi111.stderr:++ sudo rbd device map rbd/image7 2023-10-03T17:36:24.059 INFO:tasks.workunit.client.0.smithi111.stderr:+ DEVS[$i]=/dev/rbd6 2023-10-03T17:36:24.059 INFO:tasks.workunit.client.0.smithi111.stderr:+ (( i++ )) 2023-10-03T17:36:24.059 INFO:tasks.workunit.client.0.smithi111.stderr:+ (( i <= 20 )) 2023-10-03T17:36:24.059 INFO:tasks.workunit.client.0.smithi111.stderr:+ fio --name=fiotest --filename=/dev/rbd6 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-03T17:36:24.059 INFO:tasks.workunit.client.0.smithi111.stderr:++ sudo rbd device map rbd/image8 2023-10-03T17:36:24.172 INFO:tasks.workunit.client.0.smithi111.stderr:+ DEVS[$i]=/dev/rbd7 2023-10-03T17:36:24.172 INFO:tasks.workunit.client.0.smithi111.stderr:+ (( i++ )) 2023-10-03T17:36:24.172 INFO:tasks.workunit.client.0.smithi111.stderr:+ (( i <= 20 )) 2023-10-03T17:36:24.173 INFO:tasks.workunit.client.0.smithi111.stderr:+ fio --name=fiotest --filename=/dev/rbd7 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-03T17:36:24.173 INFO:tasks.workunit.client.0.smithi111.stderr:++ sudo rbd device map rbd/image9 2023-10-03T17:36:24.289 INFO:tasks.workunit.client.0.smithi111.stderr:+ DEVS[$i]=/dev/rbd8 2023-10-03T17:36:24.289 INFO:tasks.workunit.client.0.smithi111.stderr:+ (( i++ )) 2023-10-03T17:36:24.289 INFO:tasks.workunit.client.0.smithi111.stderr:+ (( i <= 20 )) 2023-10-03T17:36:24.289 INFO:tasks.workunit.client.0.smithi111.stderr:+ fio --name=fiotest --filename=/dev/rbd8 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-03T17:36:24.298 INFO:tasks.workunit.client.0.smithi111.stderr:++ sudo rbd device map rbd/image10 2023-10-03T17:36:24.393 INFO:tasks.workunit.client.0.smithi111.stderr:+ DEVS[$i]=/dev/rbd9 2023-10-03T17:36:24.394 INFO:tasks.workunit.client.0.smithi111.stderr:+ (( i++ )) 2023-10-03T17:36:24.394 INFO:tasks.workunit.client.0.smithi111.stderr:+ (( i <= 20 )) 2023-10-03T17:36:24.394 INFO:tasks.workunit.client.0.smithi111.stderr:++ sudo rbd device map rbd/image11 2023-10-03T17:36:24.394 INFO:tasks.workunit.client.0.smithi111.stderr:+ fio --name=fiotest --filename=/dev/rbd9 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-03T17:36:24.523 INFO:tasks.workunit.client.0.smithi111.stderr:+ DEVS[$i]=/dev/rbd10 2023-10-03T17:36:24.523 INFO:tasks.workunit.client.0.smithi111.stderr:+ (( i++ )) 2023-10-03T17:36:24.523 INFO:tasks.workunit.client.0.smithi111.stderr:+ (( i <= 20 )) 2023-10-03T17:36:24.523 INFO:tasks.workunit.client.0.smithi111.stderr:+ fio --name=fiotest --filename=/dev/rbd10 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-03T17:36:24.533 INFO:tasks.workunit.client.0.smithi111.stderr:++ sudo rbd device map rbd/image12 2023-10-03T17:36:24.699 INFO:tasks.workunit.client.0.smithi111.stderr:+ DEVS[$i]=/dev/rbd11 2023-10-03T17:36:24.699 INFO:tasks.workunit.client.0.smithi111.stderr:+ (( i++ )) 2023-10-03T17:36:24.699 INFO:tasks.workunit.client.0.smithi111.stderr:+ (( i <= 20 )) 2023-10-03T17:36:24.699 INFO:tasks.workunit.client.0.smithi111.stderr:+ fio --name=fiotest --filename=/dev/rbd11 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-03T17:36:24.699 INFO:tasks.workunit.client.0.smithi111.stderr:++ sudo rbd device map rbd/image13 2023-10-03T17:36:24.853 INFO:tasks.workunit.client.0.smithi111.stderr:+ DEVS[$i]=/dev/rbd12 2023-10-03T17:36:24.853 INFO:tasks.workunit.client.0.smithi111.stderr:+ (( i++ )) 2023-10-03T17:36:24.853 INFO:tasks.workunit.client.0.smithi111.stderr:+ (( i <= 20 )) 2023-10-03T17:36:24.853 INFO:tasks.workunit.client.0.smithi111.stderr:+ fio --name=fiotest --filename=/dev/rbd12 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-03T17:36:24.853 INFO:tasks.workunit.client.0.smithi111.stderr:++ sudo rbd device map rbd/image14 2023-10-03T17:36:24.987 INFO:tasks.workunit.client.0.smithi111.stderr:+ DEVS[$i]=/dev/rbd13 2023-10-03T17:36:24.987 INFO:tasks.workunit.client.0.smithi111.stderr:+ (( i++ )) 2023-10-03T17:36:24.987 INFO:tasks.workunit.client.0.smithi111.stderr:+ (( i <= 20 )) 2023-10-03T17:36:24.987 INFO:tasks.workunit.client.0.smithi111.stderr:+ fio --name=fiotest --filename=/dev/rbd13 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-03T17:36:24.987 INFO:tasks.workunit.client.0.smithi111.stderr:++ sudo rbd device map rbd/image15 2023-10-03T17:36:25.112 INFO:tasks.workunit.client.0.smithi111.stderr:+ DEVS[$i]=/dev/rbd14 2023-10-03T17:36:25.113 INFO:tasks.workunit.client.0.smithi111.stderr:+ (( i++ )) 2023-10-03T17:36:25.113 INFO:tasks.workunit.client.0.smithi111.stderr:+ (( i <= 20 )) 2023-10-03T17:36:25.113 INFO:tasks.workunit.client.0.smithi111.stderr:+ fio --name=fiotest --filename=/dev/rbd14 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-03T17:36:25.114 INFO:tasks.workunit.client.0.smithi111.stderr:++ sudo rbd device map rbd/image16 2023-10-03T17:36:25.263 INFO:tasks.workunit.client.0.smithi111.stderr:+ DEVS[$i]=/dev/rbd15 2023-10-03T17:36:25.263 INFO:tasks.workunit.client.0.smithi111.stderr:+ (( i++ )) 2023-10-03T17:36:25.263 INFO:tasks.workunit.client.0.smithi111.stderr:+ (( i <= 20 )) 2023-10-03T17:36:25.263 INFO:tasks.workunit.client.0.smithi111.stderr:+ fio --name=fiotest --filename=/dev/rbd15 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-03T17:36:25.263 INFO:tasks.workunit.client.0.smithi111.stderr:++ sudo rbd device map rbd/image17 2023-10-03T17:36:25.430 INFO:tasks.workunit.client.0.smithi111.stderr:+ DEVS[$i]=/dev/rbd16 2023-10-03T17:36:25.430 INFO:tasks.workunit.client.0.smithi111.stderr:+ (( i++ )) 2023-10-03T17:36:25.430 INFO:tasks.workunit.client.0.smithi111.stderr:+ (( i <= 20 )) 2023-10-03T17:36:25.430 INFO:tasks.workunit.client.0.smithi111.stderr:+ fio --name=fiotest --filename=/dev/rbd16 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-03T17:36:25.430 INFO:tasks.workunit.client.0.smithi111.stderr:++ sudo rbd device map rbd/image18 2023-10-03T17:36:25.584 INFO:tasks.workunit.client.0.smithi111.stderr:+ DEVS[$i]=/dev/rbd17 2023-10-03T17:36:25.584 INFO:tasks.workunit.client.0.smithi111.stderr:+ (( i++ )) 2023-10-03T17:36:25.584 INFO:tasks.workunit.client.0.smithi111.stderr:+ (( i <= 20 )) 2023-10-03T17:36:25.584 INFO:tasks.workunit.client.0.smithi111.stderr:+ fio --name=fiotest --filename=/dev/rbd17 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-03T17:36:25.590 INFO:tasks.workunit.client.0.smithi111.stderr:++ sudo rbd device map rbd/image19 2023-10-03T17:36:25.727 INFO:tasks.workunit.client.0.smithi111.stderr:+ DEVS[$i]=/dev/rbd18 2023-10-03T17:36:25.727 INFO:tasks.workunit.client.0.smithi111.stderr:+ (( i++ )) 2023-10-03T17:36:25.727 INFO:tasks.workunit.client.0.smithi111.stderr:+ (( i <= 20 )) 2023-10-03T17:36:25.727 INFO:tasks.workunit.client.0.smithi111.stderr:+ fio --name=fiotest --filename=/dev/rbd18 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-03T17:36:25.732 INFO:tasks.workunit.client.0.smithi111.stderr:++ sudo rbd device map rbd/image20 2023-10-03T17:36:25.892 INFO:tasks.workunit.client.0.smithi111.stderr:+ DEVS[$i]=/dev/rbd19 2023-10-03T17:36:25.893 INFO:tasks.workunit.client.0.smithi111.stderr:+ (( i++ )) 2023-10-03T17:36:25.893 INFO:tasks.workunit.client.0.smithi111.stderr:+ (( i <= 20 )) 2023-10-03T17:36:25.893 INFO:tasks.workunit.client.0.smithi111.stderr:+ fio --name=fiotest --filename=/dev/rbd19 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-03T17:36:25.893 INFO:tasks.workunit.client.0.smithi111.stderr:++ date +%s 2023-10-03T17:36:25.894 INFO:tasks.workunit.client.0.smithi111.stderr:+ CURRENT_TIME=1696354585 2023-10-03T17:36:25.894 INFO:tasks.workunit.client.0.smithi111.stderr:+ END_TIME=1696358185 2023-10-03T17:36:25.894 INFO:tasks.workunit.client.0.smithi111.stderr:+ PREV_CLIENT_ADDR= 2023-10-03T17:36:25.894 INFO:tasks.workunit.client.0.smithi111.stderr:+ CLIENT_ADDR= 2023-10-03T17:36:25.894 INFO:tasks.workunit.client.0.smithi111.stderr:+ [[ 1696354585 -le 1696358185 ]] 2023-10-03T17:36:25.894 INFO:tasks.workunit.client.0.smithi111.stderr:+ [[ '' != '' ]] 2023-10-03T17:36:25.894 INFO:tasks.workunit.client.0.smithi111.stderr:+ sleep 10 2023-10-03T17:36:35.895 INFO:tasks.workunit.client.0.smithi111.stderr:++ ceph mgr dump 2023-10-03T17:36:35.895 INFO:tasks.workunit.client.0.smithi111.stderr:++ jq '.active_clients[]' 2023-10-03T17:36:35.905 INFO:tasks.workunit.client.0.smithi111.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T17:36:35.910 INFO:tasks.workunit.client.0.smithi111.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T17:36:36.824 INFO:tasks.workunit.client.0.smithi111.stderr:+ CLIENT_ADDR=172.21.15.111:0/3950478362 2023-10-03T17:36:36.825 INFO:tasks.workunit.client.0.smithi111.stderr:++ date +%s 2023-10-03T17:36:36.827 INFO:tasks.workunit.client.0.smithi111.stderr:+ CURRENT_TIME=1696354596 2023-10-03T17:36:36.827 INFO:tasks.workunit.client.0.smithi111.stderr:+ [[ 1696354596 -le 1696358185 ]] 2023-10-03T17:36:36.828 INFO:tasks.workunit.client.0.smithi111.stderr:+ [[ 172.21.15.111:0/3950478362 != '' ]] 2023-10-03T17:36:36.828 INFO:tasks.workunit.client.0.smithi111.stderr:+ ceph osd blocklist add 172.21.15.111:0/3950478362 2023-10-03T17:36:39.171 INFO:tasks.workunit.client.0.smithi111.stderr:blocklisting 172.21.15.111:0/3950478362 until 2023-10-03T18:36:38.358509+0000 (3600 sec) 2023-10-03T17:36:39.201 INFO:tasks.workunit.client.0.smithi111.stderr:+ ceph osd blocklist ls 2023-10-03T17:36:39.201 INFO:tasks.workunit.client.0.smithi111.stderr:+ grep -q 172.21.15.111:0/3950478362 2023-10-03T17:36:39.905 INFO:tasks.workunit.client.0.smithi111.stderr:listed 1 entries 2023-10-03T17:36:39.933 INFO:tasks.workunit.client.0.smithi111.stderr:+ PREV_CLIENT_ADDR=172.21.15.111:0/3950478362 2023-10-03T17:36:39.933 INFO:tasks.workunit.client.0.smithi111.stderr:+ sleep 10 2023-10-03T17:36:49.935 INFO:tasks.workunit.client.0.smithi111.stderr:++ jq '.active_clients[]' 2023-10-03T17:36:49.941 INFO:tasks.workunit.client.0.smithi111.stderr:++ ceph mgr dump 2023-10-03T17:36:49.950 INFO:tasks.workunit.client.0.smithi111.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T17:36:49.958 INFO:tasks.workunit.client.0.smithi111.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T17:36:50.914 INFO:tasks.workunit.client.0.smithi111.stderr:+ CLIENT_ADDR=172.21.15.111:0/3950478362 2023-10-03T17:36:50.914 INFO:tasks.workunit.client.0.smithi111.stderr:++ date +%s 2023-10-03T17:36:50.915 INFO:tasks.workunit.client.0.smithi111.stderr:+ CURRENT_TIME=1696354610 2023-10-03T17:36:50.915 INFO:tasks.workunit.client.0.smithi111.stderr:+ [[ 1696354610 -le 1696358185 ]] 2023-10-03T17:36:50.915 INFO:tasks.workunit.client.0.smithi111.stderr:+ [[ 172.21.15.111:0/3950478362 != \1\7\2\.\2\1\.\1\5\.\1\1\1\:\0\/\3\9\5\0\4\7\8\3\6\2 ]] 2023-10-03T17:36:50.915 INFO:tasks.workunit.client.0.smithi111.stderr:+ sleep 10 2023-10-03T17:36:57.190 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:36:57.187+0000 7f09e4851640 -1 reset not still connected to 0x557d75f2b2b0 2023-10-03T17:36:57.190 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:36:57.187+0000 7f09e4851640 -1 reset not still connected to 0x557d76253d40 2023-10-03T17:36:57.190 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:36:57.187+0000 7f09e4851640 -1 reset not still connected to 0x557d76621040 2023-10-03T17:36:57.190 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:36:57.187+0000 7f09e4851640 -1 reset not still connected to 0x557d7664e750 2023-10-03T17:36:57.191 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:36:57.187+0000 7f09e4851640 -1 reset not still connected to 0x557d7664ea90 2023-10-03T17:36:57.191 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:36:57.187+0000 7f09e4851640 -1 reset not still connected to 0x557d7664f380 2023-10-03T17:36:57.191 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:36:57.187+0000 7f09e4851640 -1 reset not still connected to 0x557d7664f930 2023-10-03T17:36:57.191 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:36:57.187+0000 7f09e4851640 -1 reset not still connected to 0x557d76681ba0 2023-10-03T17:36:57.191 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:36:57.187+0000 7f09e4851640 -1 reset not still connected to 0x557d766a40d0 2023-10-03T17:36:57.191 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:36:57.187+0000 7f09e4851640 -1 reset not still connected to 0x557d766b56c0 2023-10-03T17:36:57.191 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:36:57.187+0000 7f09e4851640 -1 reset not still connected to 0x557d76eca270 2023-10-03T17:36:57.191 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:36:57.187+0000 7f09e4851640 -1 reset not still connected to 0x557d76f93ba0 2023-10-03T17:37:00.006 INFO:tasks.ceph.mgr.x.smithi111.stderr:2023-10-03T17:37:00.006+0000 7fe483b87640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:37:00.006 INFO:tasks.ceph.mgr.x.smithi111.stderr:2023-10-03T17:37:00.006+0000 7fe483b87640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:37:00.006 INFO:tasks.ceph.mgr.x.smithi111.stderr:2023-10-03T17:37:00.006+0000 7fe483b87640 -1 librbd::ImageState: 0x55f87dccbf80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:37:00.007 INFO:tasks.ceph.mgr.x.smithi111.stderr:2023-10-03T17:37:00.006+0000 7fe483b87640 -1 librbd::ImageState: 0x55f87e0b3b80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:37:00.012 INFO:tasks.ceph.mgr.x.smithi111.stderr:2023-10-03T17:37:00.010+0000 7fe483b87640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:37:00.012 INFO:tasks.ceph.mgr.x.smithi111.stderr:2023-10-03T17:37:00.010+0000 7fe483b87640 -1 librbd::ImageState: 0x55f87e00ab80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:37:00.019 INFO:tasks.ceph.mgr.x.smithi111.stderr:2023-10-03T17:37:00.018+0000 7fe483b87640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:37:00.019 INFO:tasks.ceph.mgr.x.smithi111.stderr:2023-10-03T17:37:00.018+0000 7fe483b87640 -1 librbd::ImageState: 0x55f87dccd200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:37:00.022 INFO:tasks.ceph.mgr.x.smithi111.stderr:2023-10-03T17:37:00.022+0000 7fe483b87640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:37:00.022 INFO:tasks.ceph.mgr.x.smithi111.stderr:2023-10-03T17:37:00.022+0000 7fe483b87640 -1 librbd::ImageState: 0x55f87dcccb00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:37:00.035 INFO:tasks.ceph.mgr.x.smithi111.stderr:2023-10-03T17:37:00.034+0000 7fe483b87640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:37:00.035 INFO:tasks.ceph.mgr.x.smithi111.stderr:2023-10-03T17:37:00.034+0000 7fe483b87640 -1 librbd::ImageState: 0x55f87e0b3d00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:37:00.042 INFO:tasks.ceph.mgr.x.smithi111.stderr:2023-10-03T17:37:00.042+0000 7fe483b87640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:37:00.042 INFO:tasks.ceph.mgr.x.smithi111.stderr:2023-10-03T17:37:00.042+0000 7fe483b87640 -1 librbd::ImageState: 0x55f87e0b3900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:37:00.047 INFO:tasks.ceph.mgr.x.smithi111.stderr:2023-10-03T17:37:00.046+0000 7fe483b87640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:37:00.047 INFO:tasks.ceph.mgr.x.smithi111.stderr:2023-10-03T17:37:00.046+0000 7fe483b87640 -1 librbd::ImageState: 0x55f87dccb100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:37:00.060 INFO:tasks.ceph.mgr.x.smithi111.stderr:2023-10-03T17:37:00.058+0000 7fe483b87640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:37:00.060 INFO:tasks.ceph.mgr.x.smithi111.stderr:2023-10-03T17:37:00.058+0000 7fe483b87640 -1 librbd::ImageState: 0x55f87689e280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:37:00.070 INFO:tasks.ceph.mgr.x.smithi111.stderr:2023-10-03T17:37:00.066+0000 7fe483b87640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:37:00.070 INFO:tasks.ceph.mgr.x.smithi111.stderr:2023-10-03T17:37:00.066+0000 7fe483b87640 -1 librbd::ImageState: 0x55f87e00ab80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:37:00.079 INFO:tasks.ceph.mgr.x.smithi111.stderr:2023-10-03T17:37:00.078+0000 7fe483b87640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:37:00.079 INFO:tasks.ceph.mgr.x.smithi111.stderr:2023-10-03T17:37:00.078+0000 7fe483b87640 -1 librbd::ImageState: 0x55f87dccbd00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:37:00.081 INFO:tasks.ceph.mgr.x.smithi111.stderr:2023-10-03T17:37:00.078+0000 7fe483b87640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:37:00.081 INFO:tasks.ceph.mgr.x.smithi111.stderr:2023-10-03T17:37:00.078+0000 7fe483b87640 -1 librbd::ImageState: 0x55f87e0b3700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:37:00.084 INFO:tasks.ceph.mgr.x.smithi111.stderr:2023-10-03T17:37:00.082+0000 7fe483b87640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:37:00.084 INFO:tasks.ceph.mgr.x.smithi111.stderr:2023-10-03T17:37:00.082+0000 7fe483b87640 -1 librbd::ImageState: 0x55f87dbeab00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:37:00.093 INFO:tasks.ceph.mgr.x.smithi111.stderr:2023-10-03T17:37:00.090+0000 7fe483b87640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:37:00.094 INFO:tasks.ceph.mgr.x.smithi111.stderr:2023-10-03T17:37:00.090+0000 7fe483b87640 -1 librbd::ImageState: 0x55f87dccca80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:37:00.097 INFO:tasks.ceph.mgr.x.smithi111.stderr:2023-10-03T17:37:00.094+0000 7fe483b87640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:37:00.097 INFO:tasks.ceph.mgr.x.smithi111.stderr:2023-10-03T17:37:00.094+0000 7fe483b87640 -1 librbd::ImageState: 0x55f87dcccb80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:37:00.101 INFO:tasks.ceph.mgr.x.smithi111.stderr:2023-10-03T17:37:00.098+0000 7fe483b87640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:37:00.101 INFO:tasks.ceph.mgr.x.smithi111.stderr:2023-10-03T17:37:00.098+0000 7fe483b87640 -1 librbd::ImageState: 0x55f87dccde00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:37:00.105 INFO:tasks.ceph.mgr.x.smithi111.stderr:2023-10-03T17:37:00.102+0000 7fe483b87640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:37:00.105 INFO:tasks.ceph.mgr.x.smithi111.stderr:2023-10-03T17:37:00.102+0000 7fe483b87640 -1 librbd::ImageState: 0x55f87e0b3800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:37:00.109 INFO:tasks.ceph.mgr.x.smithi111.stderr:2023-10-03T17:37:00.106+0000 7fe483b87640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:37:00.109 INFO:tasks.ceph.mgr.x.smithi111.stderr:2023-10-03T17:37:00.106+0000 7fe483b87640 -1 librbd::ImageState: 0x55f87db9ff00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:37:00.112 INFO:tasks.ceph.mgr.x.smithi111.stderr:2023-10-03T17:37:00.110+0000 7fe483b87640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:37:00.113 INFO:tasks.ceph.mgr.x.smithi111.stderr:2023-10-03T17:37:00.110+0000 7fe483b87640 -1 librbd::ImageState: 0x55f87e0b3b80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:37:00.117 INFO:tasks.ceph.mgr.x.smithi111.stderr:2023-10-03T17:37:00.114+0000 7fe483b87640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:37:00.117 INFO:tasks.ceph.mgr.x.smithi111.stderr:2023-10-03T17:37:00.114+0000 7fe483b87640 -1 librbd::ImageState: 0x55f87e0b3900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:37:00.917 INFO:tasks.workunit.client.0.smithi111.stderr:++ ceph mgr dump 2023-10-03T17:37:00.917 INFO:tasks.workunit.client.0.smithi111.stderr:++ jq '.active_clients[]' 2023-10-03T17:37:00.917 INFO:tasks.workunit.client.0.smithi111.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T17:37:00.917 INFO:tasks.workunit.client.0.smithi111.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T17:37:01.792 INFO:tasks.workunit.client.0.smithi111.stderr:+ CLIENT_ADDR=172.21.15.111:0/3950478362 2023-10-03T17:37:01.792 INFO:tasks.workunit.client.0.smithi111.stderr:++ date +%s 2023-10-03T17:37:01.793 INFO:tasks.workunit.client.0.smithi111.stderr:+ CURRENT_TIME=1696354621 2023-10-03T17:37:01.793 INFO:tasks.workunit.client.0.smithi111.stderr:+ [[ 1696354621 -le 1696358185 ]] 2023-10-03T17:37:01.793 INFO:tasks.workunit.client.0.smithi111.stderr:+ [[ 172.21.15.111:0/3950478362 != \1\7\2\.\2\1\.\1\5\.\1\1\1\:\0\/\3\9\5\0\4\7\8\3\6\2 ]] 2023-10-03T17:37:01.793 INFO:tasks.workunit.client.0.smithi111.stderr:+ sleep 10 2023-10-03T17:37:11.795 INFO:tasks.workunit.client.0.smithi111.stderr:++ ceph mgr dump 2023-10-03T17:37:11.795 INFO:tasks.workunit.client.0.smithi111.stderr:++ jq '.active_clients[]' 2023-10-03T17:37:11.795 INFO:tasks.workunit.client.0.smithi111.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T17:37:11.797 INFO:tasks.workunit.client.0.smithi111.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T17:37:12.718 INFO:tasks.workunit.client.0.smithi111.stderr:+ CLIENT_ADDR=172.21.15.111:0/3950478362 2023-10-03T17:37:12.718 INFO:tasks.workunit.client.0.smithi111.stderr:++ date +%s 2023-10-03T17:37:12.725 INFO:tasks.workunit.client.0.smithi111.stderr:+ CURRENT_TIME=1696354632 2023-10-03T17:37:12.725 INFO:tasks.workunit.client.0.smithi111.stderr:+ [[ 1696354632 -le 1696358185 ]] 2023-10-03T17:37:12.725 INFO:tasks.workunit.client.0.smithi111.stderr:+ [[ 172.21.15.111:0/3950478362 != \1\7\2\.\2\1\.\1\5\.\1\1\1\:\0\/\3\9\5\0\4\7\8\3\6\2 ]] 2023-10-03T17:37:12.725 INFO:tasks.workunit.client.0.smithi111.stderr:+ sleep 10 2023-10-03T17:37:22.732 INFO:tasks.workunit.client.0.smithi111.stderr:++ ceph mgr dump 2023-10-03T17:37:22.733 INFO:tasks.workunit.client.0.smithi111.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T17:37:22.733 INFO:tasks.workunit.client.0.smithi111.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T17:37:22.746 INFO:tasks.workunit.client.0.smithi111.stderr:++ jq '.active_clients[]' 2023-10-03T17:37:23.608 INFO:tasks.workunit.client.0.smithi111.stderr:+ CLIENT_ADDR=172.21.15.111:0/3950478362 2023-10-03T17:37:23.608 INFO:tasks.workunit.client.0.smithi111.stderr:++ date +%s 2023-10-03T17:37:23.609 INFO:tasks.workunit.client.0.smithi111.stderr:+ CURRENT_TIME=1696354643 2023-10-03T17:37:23.609 INFO:tasks.workunit.client.0.smithi111.stderr:+ [[ 1696354643 -le 1696358185 ]] 2023-10-03T17:37:23.609 INFO:tasks.workunit.client.0.smithi111.stderr:+ [[ 172.21.15.111:0/3950478362 != \1\7\2\.\2\1\.\1\5\.\1\1\1\:\0\/\3\9\5\0\4\7\8\3\6\2 ]] 2023-10-03T17:37:23.609 INFO:tasks.workunit.client.0.smithi111.stderr:+ sleep 10 2023-10-03T17:37:32.404 INFO:tasks.ceph.osd.0.smithi111.stderr:2023-10-03T17:37:32.402+0000 7ffbac289640 -1 reset not still connected to 0x5563a7ff65b0 2023-10-03T17:37:32.404 INFO:tasks.ceph.osd.0.smithi111.stderr:2023-10-03T17:37:32.402+0000 7ffbac289640 -1 reset not still connected to 0x5563a7ff68f0 2023-10-03T17:37:32.405 INFO:tasks.ceph.osd.0.smithi111.stderr:2023-10-03T17:37:32.402+0000 7ffbac289640 -1 reset not still connected to 0x5563a838e1a0 2023-10-03T17:37:32.405 INFO:tasks.ceph.osd.0.smithi111.stderr:2023-10-03T17:37:32.402+0000 7ffbac289640 -1 reset not still connected to 0x5563a851aea0 2023-10-03T17:37:33.611 INFO:tasks.workunit.client.0.smithi111.stderr:++ ceph mgr dump 2023-10-03T17:37:33.611 INFO:tasks.workunit.client.0.smithi111.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T17:37:33.613 INFO:tasks.workunit.client.0.smithi111.stderr:++ jq '.active_clients[]' 2023-10-03T17:37:33.613 INFO:tasks.workunit.client.0.smithi111.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T17:37:34.580 INFO:tasks.workunit.client.0.smithi111.stderr:+ CLIENT_ADDR=172.21.15.111:0/3950478362 2023-10-03T17:37:34.580 INFO:tasks.workunit.client.0.smithi111.stderr:++ date +%s 2023-10-03T17:37:34.582 INFO:tasks.workunit.client.0.smithi111.stderr:+ CURRENT_TIME=1696354654 2023-10-03T17:37:34.582 INFO:tasks.workunit.client.0.smithi111.stderr:+ [[ 1696354654 -le 1696358185 ]] 2023-10-03T17:37:34.582 INFO:tasks.workunit.client.0.smithi111.stderr:+ [[ 172.21.15.111:0/3950478362 != \1\7\2\.\2\1\.\1\5\.\1\1\1\:\0\/\3\9\5\0\4\7\8\3\6\2 ]] 2023-10-03T17:37:34.582 INFO:tasks.workunit.client.0.smithi111.stderr:+ sleep 10 2023-10-03T17:37:44.584 INFO:tasks.workunit.client.0.smithi111.stderr:++ jq '.active_clients[]' 2023-10-03T17:37:44.585 INFO:tasks.workunit.client.0.smithi111.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T17:37:44.593 INFO:tasks.workunit.client.0.smithi111.stderr:++ ceph mgr dump 2023-10-03T17:37:44.595 INFO:tasks.workunit.client.0.smithi111.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T17:37:45.550 INFO:tasks.workunit.client.0.smithi111.stderr:+ CLIENT_ADDR=172.21.15.111:0/262203629 2023-10-03T17:37:45.550 INFO:tasks.workunit.client.0.smithi111.stderr:++ date +%s 2023-10-03T17:37:45.551 INFO:tasks.workunit.client.0.smithi111.stderr:+ CURRENT_TIME=1696354665 2023-10-03T17:37:45.551 INFO:tasks.workunit.client.0.smithi111.stderr:+ [[ 1696354665 -le 1696358185 ]] 2023-10-03T17:37:45.551 INFO:tasks.workunit.client.0.smithi111.stderr:+ [[ 172.21.15.111:0/262203629 != \1\7\2\.\2\1\.\1\5\.\1\1\1\:\0\/\3\9\5\0\4\7\8\3\6\2 ]] 2023-10-03T17:37:45.551 INFO:tasks.workunit.client.0.smithi111.stderr:+ ceph osd blocklist add 172.21.15.111:0/262203629 2023-10-03T17:37:47.225 INFO:tasks.workunit.client.0.smithi111.stderr:blocklisting 172.21.15.111:0/262203629 until 2023-10-03T18:37:46.493199+0000 (3600 sec) 2023-10-03T17:37:47.268 INFO:tasks.workunit.client.0.smithi111.stderr:+ ceph osd blocklist ls 2023-10-03T17:37:47.268 INFO:tasks.workunit.client.0.smithi111.stderr:+ grep -q 172.21.15.111:0/262203629 2023-10-03T17:37:47.973 INFO:tasks.workunit.client.0.smithi111.stderr:listed 2 entries 2023-10-03T17:37:48.003 INFO:tasks.workunit.client.0.smithi111.stderr:+ PREV_CLIENT_ADDR=172.21.15.111:0/262203629 2023-10-03T17:37:48.003 INFO:tasks.workunit.client.0.smithi111.stderr:+ sleep 10 2023-10-03T17:37:58.006 INFO:tasks.workunit.client.0.smithi111.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T17:37:58.012 INFO:tasks.workunit.client.0.smithi111.stderr:++ ceph mgr dump 2023-10-03T17:37:58.019 INFO:tasks.workunit.client.0.smithi111.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T17:37:58.030 INFO:tasks.workunit.client.0.smithi111.stderr:++ jq '.active_clients[]' 2023-10-03T17:37:58.999 INFO:tasks.workunit.client.0.smithi111.stderr:+ CLIENT_ADDR=172.21.15.111:0/262203629 2023-10-03T17:37:58.999 INFO:tasks.workunit.client.0.smithi111.stderr:++ date +%s 2023-10-03T17:37:59.000 INFO:tasks.workunit.client.0.smithi111.stderr:+ CURRENT_TIME=1696354679 2023-10-03T17:37:59.000 INFO:tasks.workunit.client.0.smithi111.stderr:+ [[ 1696354679 -le 1696358185 ]] 2023-10-03T17:37:59.000 INFO:tasks.workunit.client.0.smithi111.stderr:+ [[ 172.21.15.111:0/262203629 != \1\7\2\.\2\1\.\1\5\.\1\1\1\:\0\/\2\6\2\2\0\3\6\2\9 ]] 2023-10-03T17:37:59.001 INFO:tasks.workunit.client.0.smithi111.stderr:+ sleep 10 2023-10-03T17:38:00.003 INFO:tasks.ceph.mgr.x.smithi111.stderr:2023-10-03T17:38:00.002+0000 7fe482b85640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:38:00.003 INFO:tasks.ceph.mgr.x.smithi111.stderr:2023-10-03T17:38:00.002+0000 7fe482b85640 -1 librbd::ImageState: 0x55f87e0b2900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:38:00.003 INFO:tasks.ceph.mgr.x.smithi111.stderr:2023-10-03T17:38:00.002+0000 7fe482b85640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:38:00.003 INFO:tasks.ceph.mgr.x.smithi111.stderr:2023-10-03T17:38:00.002+0000 7fe482b85640 -1 librbd::ImageState: 0x55f87dbeae80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:38:00.010 INFO:tasks.ceph.mgr.x.smithi111.stderr:2023-10-03T17:38:00.010+0000 7fe482b85640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:38:00.010 INFO:tasks.ceph.mgr.x.smithi111.stderr:2023-10-03T17:38:00.010+0000 7fe482b85640 -1 librbd::ImageState: 0x55f87e7b3c80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:38:00.020 INFO:tasks.ceph.mgr.x.smithi111.stderr:2023-10-03T17:38:00.018+0000 7fe482b85640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:38:00.020 INFO:tasks.ceph.mgr.x.smithi111.stderr:2023-10-03T17:38:00.018+0000 7fe482b85640 -1 librbd::ImageState: 0x55f87e00b500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:38:00.036 INFO:tasks.ceph.mgr.x.smithi111.stderr:2023-10-03T17:38:00.034+0000 7fe482b85640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:38:00.036 INFO:tasks.ceph.mgr.x.smithi111.stderr:2023-10-03T17:38:00.034+0000 7fe482b85640 -1 librbd::ImageState: 0x55f87e7b3d80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:38:00.042 INFO:tasks.ceph.mgr.x.smithi111.stderr:2023-10-03T17:38:00.042+0000 7fe482b85640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:38:00.042 INFO:tasks.ceph.mgr.x.smithi111.stderr:2023-10-03T17:38:00.042+0000 7fe482b85640 -1 librbd::ImageState: 0x55f87e7b3680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:38:00.055 INFO:tasks.ceph.mgr.x.smithi111.stderr:2023-10-03T17:38:00.054+0000 7fe482b85640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:38:00.055 INFO:tasks.ceph.mgr.x.smithi111.stderr:2023-10-03T17:38:00.054+0000 7fe482b85640 -1 librbd::ImageState: 0x55f87dbea880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:38:00.060 INFO:tasks.ceph.mgr.x.smithi111.stderr:2023-10-03T17:38:00.058+0000 7fe482b85640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:38:00.060 INFO:tasks.ceph.mgr.x.smithi111.stderr:2023-10-03T17:38:00.058+0000 7fe482b85640 -1 librbd::ImageState: 0x55f87e7b3e80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:38:00.065 INFO:tasks.ceph.mgr.x.smithi111.stderr:2023-10-03T17:38:00.066+0000 7fe482b85640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:38:00.066 INFO:tasks.ceph.mgr.x.smithi111.stderr:2023-10-03T17:38:00.066+0000 7fe482b85640 -1 librbd::ImageState: 0x55f87e7b3380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:38:00.068 INFO:tasks.ceph.mgr.x.smithi111.stderr:2023-10-03T17:38:00.066+0000 7fe482b85640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:38:00.068 INFO:tasks.ceph.mgr.x.smithi111.stderr:2023-10-03T17:38:00.066+0000 7fe483386640 -1 librbd::ImageState: 0x55f87e7b3700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:38:00.070 INFO:tasks.ceph.mgr.x.smithi111.stderr:2023-10-03T17:38:00.070+0000 7fe483386640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:38:00.070 INFO:tasks.ceph.mgr.x.smithi111.stderr:2023-10-03T17:38:00.070+0000 7fe483386640 -1 librbd::ImageState: 0x55f87e7b3a80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:38:00.074 INFO:tasks.ceph.mgr.x.smithi111.stderr:2023-10-03T17:38:00.074+0000 7fe482b85640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:38:00.074 INFO:tasks.ceph.mgr.x.smithi111.stderr:2023-10-03T17:38:00.074+0000 7fe482b85640 -1 librbd::ImageState: 0x55f87e00b500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:38:00.078 INFO:tasks.ceph.mgr.x.smithi111.stderr:2023-10-03T17:38:00.078+0000 7fe482b85640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:38:00.078 INFO:tasks.ceph.mgr.x.smithi111.stderr:2023-10-03T17:38:00.078+0000 7fe482b85640 -1 librbd::ImageState: 0x55f87dbeb680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:38:00.081 INFO:tasks.ceph.mgr.x.smithi111.stderr:2023-10-03T17:38:00.078+0000 7fe482b85640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:38:00.081 INFO:tasks.ceph.mgr.x.smithi111.stderr:2023-10-03T17:38:00.078+0000 7fe482b85640 -1 librbd::ImageState: 0x55f87e7b3680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:38:00.084 INFO:tasks.ceph.mgr.x.smithi111.stderr:2023-10-03T17:38:00.082+0000 7fe483386640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:38:00.084 INFO:tasks.ceph.mgr.x.smithi111.stderr:2023-10-03T17:38:00.082+0000 7fe483386640 -1 librbd::ImageState: 0x55f87e7b3780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:38:00.087 INFO:tasks.ceph.mgr.x.smithi111.stderr:2023-10-03T17:38:00.086+0000 7fe483386640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:38:00.087 INFO:tasks.ceph.mgr.x.smithi111.stderr:2023-10-03T17:38:00.086+0000 7fe483386640 -1 librbd::ImageState: 0x55f87e7b3380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:38:00.092 INFO:tasks.ceph.mgr.x.smithi111.stderr:2023-10-03T17:38:00.090+0000 7fe483386640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:38:00.092 INFO:tasks.ceph.mgr.x.smithi111.stderr:2023-10-03T17:38:00.090+0000 7fe483386640 -1 librbd::ImageState: 0x55f87e7b3f80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:38:00.096 INFO:tasks.ceph.mgr.x.smithi111.stderr:2023-10-03T17:38:00.094+0000 7fe483386640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:38:00.096 INFO:tasks.ceph.mgr.x.smithi111.stderr:2023-10-03T17:38:00.094+0000 7fe483386640 -1 librbd::ImageState: 0x55f87e7b3e00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:38:00.099 INFO:tasks.ceph.mgr.x.smithi111.stderr:2023-10-03T17:38:00.098+0000 7fe483386640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:38:00.099 INFO:tasks.ceph.mgr.x.smithi111.stderr:2023-10-03T17:38:00.098+0000 7fe483386640 -1 librbd::ImageState: 0x55f87dbea680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:38:00.103 INFO:tasks.ceph.mgr.x.smithi111.stderr:2023-10-03T17:38:00.102+0000 7fe483386640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:38:00.103 INFO:tasks.ceph.mgr.x.smithi111.stderr:2023-10-03T17:38:00.102+0000 7fe483386640 -1 librbd::ImageState: 0x55f87dbeb280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:38:09.009 INFO:tasks.workunit.client.0.smithi111.stderr:++ ceph mgr dump 2023-10-03T17:38:09.010 INFO:tasks.workunit.client.0.smithi111.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T17:38:09.010 INFO:tasks.workunit.client.0.smithi111.stderr:++ jq '.active_clients[]' 2023-10-03T17:38:09.016 INFO:tasks.workunit.client.0.smithi111.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T17:38:09.899 INFO:tasks.workunit.client.0.smithi111.stderr:+ CLIENT_ADDR=172.21.15.111:0/262203629 2023-10-03T17:38:09.899 INFO:tasks.workunit.client.0.smithi111.stderr:++ date +%s 2023-10-03T17:38:09.900 INFO:tasks.workunit.client.0.smithi111.stderr:+ CURRENT_TIME=1696354689 2023-10-03T17:38:09.900 INFO:tasks.workunit.client.0.smithi111.stderr:+ [[ 1696354689 -le 1696358185 ]] 2023-10-03T17:38:09.900 INFO:tasks.workunit.client.0.smithi111.stderr:+ [[ 172.21.15.111:0/262203629 != \1\7\2\.\2\1\.\1\5\.\1\1\1\:\0\/\2\6\2\2\0\3\6\2\9 ]] 2023-10-03T17:38:09.900 INFO:tasks.workunit.client.0.smithi111.stderr:+ sleep 10 2023-10-03T17:38:19.907 INFO:tasks.workunit.client.0.smithi111.stderr:++ ceph mgr dump 2023-10-03T17:38:19.909 INFO:tasks.workunit.client.0.smithi111.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T17:38:19.922 INFO:tasks.workunit.client.0.smithi111.stderr:++ jq '.active_clients[]' 2023-10-03T17:38:19.933 INFO:tasks.workunit.client.0.smithi111.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T17:38:20.947 INFO:tasks.workunit.client.0.smithi111.stderr:+ CLIENT_ADDR=172.21.15.111:0/262203629 2023-10-03T17:38:20.947 INFO:tasks.workunit.client.0.smithi111.stderr:++ date +%s 2023-10-03T17:38:20.948 INFO:tasks.workunit.client.0.smithi111.stderr:+ CURRENT_TIME=1696354700 2023-10-03T17:38:20.948 INFO:tasks.workunit.client.0.smithi111.stderr:+ [[ 1696354700 -le 1696358185 ]] 2023-10-03T17:38:20.948 INFO:tasks.workunit.client.0.smithi111.stderr:+ [[ 172.21.15.111:0/262203629 != \1\7\2\.\2\1\.\1\5\.\1\1\1\:\0\/\2\6\2\2\0\3\6\2\9 ]] 2023-10-03T17:38:20.948 INFO:tasks.workunit.client.0.smithi111.stderr:+ sleep 10 2023-10-03T17:38:30.949 INFO:tasks.workunit.client.0.smithi111.stderr:++ ceph mgr dump 2023-10-03T17:38:30.949 INFO:tasks.workunit.client.0.smithi111.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T17:38:30.952 INFO:tasks.workunit.client.0.smithi111.stderr:++ jq '.active_clients[]' 2023-10-03T17:38:30.967 INFO:tasks.workunit.client.0.smithi111.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T17:38:31.964 INFO:tasks.workunit.client.0.smithi111.stderr:+ CLIENT_ADDR=172.21.15.111:0/262203629 2023-10-03T17:38:31.964 INFO:tasks.workunit.client.0.smithi111.stderr:++ date +%s 2023-10-03T17:38:31.965 INFO:tasks.workunit.client.0.smithi111.stderr:+ CURRENT_TIME=1696354711 2023-10-03T17:38:31.965 INFO:tasks.workunit.client.0.smithi111.stderr:+ [[ 1696354711 -le 1696358185 ]] 2023-10-03T17:38:31.965 INFO:tasks.workunit.client.0.smithi111.stderr:+ [[ 172.21.15.111:0/262203629 != \1\7\2\.\2\1\.\1\5\.\1\1\1\:\0\/\2\6\2\2\0\3\6\2\9 ]] 2023-10-03T17:38:31.965 INFO:tasks.workunit.client.0.smithi111.stderr:+ sleep 10 2023-10-03T17:38:41.970 INFO:tasks.workunit.client.0.smithi111.stderr:++ ceph mgr dump 2023-10-03T17:38:41.978 INFO:tasks.workunit.client.0.smithi111.stderr:++ jq '.active_clients[]' 2023-10-03T17:38:41.983 INFO:tasks.workunit.client.0.smithi111.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T17:38:41.996 INFO:tasks.workunit.client.0.smithi111.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T17:38:42.816 INFO:tasks.workunit.client.0.smithi111.stderr:+ CLIENT_ADDR=172.21.15.111:0/262203629 2023-10-03T17:38:42.816 INFO:tasks.workunit.client.0.smithi111.stderr:++ date +%s 2023-10-03T17:38:42.817 INFO:tasks.workunit.client.0.smithi111.stderr:+ CURRENT_TIME=1696354722 2023-10-03T17:38:42.817 INFO:tasks.workunit.client.0.smithi111.stderr:+ [[ 1696354722 -le 1696358185 ]] 2023-10-03T17:38:42.817 INFO:tasks.workunit.client.0.smithi111.stderr:+ [[ 172.21.15.111:0/262203629 != \1\7\2\.\2\1\.\1\5\.\1\1\1\:\0\/\2\6\2\2\0\3\6\2\9 ]] 2023-10-03T17:38:42.817 INFO:tasks.workunit.client.0.smithi111.stderr:+ sleep 10 2023-10-03T17:38:52.818 INFO:tasks.workunit.client.0.smithi111.stderr:++ ceph mgr dump 2023-10-03T17:38:52.819 INFO:tasks.workunit.client.0.smithi111.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T17:38:52.821 INFO:tasks.workunit.client.0.smithi111.stderr:++ jq '.active_clients[]' 2023-10-03T17:38:52.827 INFO:tasks.workunit.client.0.smithi111.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T17:38:53.966 INFO:tasks.workunit.client.0.smithi111.stderr:+ CLIENT_ADDR=172.21.15.111:0/4228986752 2023-10-03T17:38:53.966 INFO:tasks.workunit.client.0.smithi111.stderr:++ date +%s 2023-10-03T17:38:53.967 INFO:tasks.workunit.client.0.smithi111.stderr:+ CURRENT_TIME=1696354733 2023-10-03T17:38:53.967 INFO:tasks.workunit.client.0.smithi111.stderr:+ [[ 1696354733 -le 1696358185 ]] 2023-10-03T17:38:53.967 INFO:tasks.workunit.client.0.smithi111.stderr:+ [[ 172.21.15.111:0/4228986752 != \1\7\2\.\2\1\.\1\5\.\1\1\1\:\0\/\2\6\2\2\0\3\6\2\9 ]] 2023-10-03T17:38:53.967 INFO:tasks.workunit.client.0.smithi111.stderr:+ ceph osd blocklist add 172.21.15.111:0/4228986752 2023-10-03T17:38:56.217 INFO:tasks.workunit.client.0.smithi111.stderr:blocklisting 172.21.15.111:0/4228986752 until 2023-10-03T18:38:55.415340+0000 (3600 sec) 2023-10-03T17:38:56.254 INFO:tasks.workunit.client.0.smithi111.stderr:+ grep -q 172.21.15.111:0/4228986752 2023-10-03T17:38:56.255 INFO:tasks.workunit.client.0.smithi111.stderr:+ ceph osd blocklist ls 2023-10-03T17:38:56.982 INFO:tasks.workunit.client.0.smithi111.stderr:listed 3 entries 2023-10-03T17:38:57.011 INFO:tasks.workunit.client.0.smithi111.stderr:+ PREV_CLIENT_ADDR=172.21.15.111:0/4228986752 2023-10-03T17:38:57.011 INFO:tasks.workunit.client.0.smithi111.stderr:+ sleep 10 2023-10-03T17:39:00.002 INFO:tasks.ceph.mgr.x.smithi111.stderr:2023-10-03T17:39:00.005+0000 7fe483b87640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:39:00.003 INFO:tasks.ceph.mgr.x.smithi111.stderr:2023-10-03T17:39:00.005+0000 7fe483b87640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:39:00.003 INFO:tasks.ceph.mgr.x.smithi111.stderr:2023-10-03T17:39:00.005+0000 7fe483b87640 -1 librbd::ImageState: 0x55f87e868000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:39:00.003 INFO:tasks.ceph.mgr.x.smithi111.stderr:2023-10-03T17:39:00.005+0000 7fe483b87640 -1 librbd::ImageState: 0x55f87e869980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:39:00.005 INFO:tasks.ceph.mgr.x.smithi111.stderr:2023-10-03T17:39:00.005+0000 7fe483b87640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:39:00.005 INFO:tasks.ceph.mgr.x.smithi111.stderr:2023-10-03T17:39:00.005+0000 7fe483b87640 -1 librbd::ImageState: 0x55f87e869b80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:39:00.012 INFO:tasks.ceph.mgr.x.smithi111.stderr:2023-10-03T17:39:00.013+0000 7fe483b87640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:39:00.012 INFO:tasks.ceph.mgr.x.smithi111.stderr:2023-10-03T17:39:00.013+0000 7fe483b87640 -1 librbd::ImageState: 0x55f87e869d80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:39:00.018 INFO:tasks.ceph.mgr.x.smithi111.stderr:2023-10-03T17:39:00.021+0000 7fe483b87640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:39:00.018 INFO:tasks.ceph.mgr.x.smithi111.stderr:2023-10-03T17:39:00.021+0000 7fe483b87640 -1 librbd::ImageState: 0x55f87e869f80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:39:00.022 INFO:tasks.ceph.mgr.x.smithi111.stderr:2023-10-03T17:39:00.025+0000 7fe483b87640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:39:00.023 INFO:tasks.ceph.mgr.x.smithi111.stderr:2023-10-03T17:39:00.025+0000 7fe483b87640 -1 librbd::ImageState: 0x55f87dbeb680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:39:00.026 INFO:tasks.ceph.mgr.x.smithi111.stderr:2023-10-03T17:39:00.029+0000 7fe483b87640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:39:00.026 INFO:tasks.ceph.mgr.x.smithi111.stderr:2023-10-03T17:39:00.029+0000 7fe483b87640 -1 librbd::ImageState: 0x55f87e868700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:39:00.029 INFO:tasks.ceph.mgr.x.smithi111.stderr:2023-10-03T17:39:00.029+0000 7fe483b87640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:39:00.029 INFO:tasks.ceph.mgr.x.smithi111.stderr:2023-10-03T17:39:00.029+0000 7fe483b87640 -1 librbd::ImageState: 0x55f87e00ae00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:39:00.032 INFO:tasks.ceph.mgr.x.smithi111.stderr:2023-10-03T17:39:00.033+0000 7fe483b87640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:39:00.032 INFO:tasks.ceph.mgr.x.smithi111.stderr:2023-10-03T17:39:00.033+0000 7fe483b87640 -1 librbd::ImageState: 0x55f87e868880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:39:00.036 INFO:tasks.ceph.mgr.x.smithi111.stderr:2023-10-03T17:39:00.037+0000 7fe483b87640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:39:00.036 INFO:tasks.ceph.mgr.x.smithi111.stderr:2023-10-03T17:39:00.037+0000 7fe483b87640 -1 librbd::ImageState: 0x55f87e868980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:39:00.039 INFO:tasks.ceph.mgr.x.smithi111.stderr:2023-10-03T17:39:00.041+0000 7fe483b87640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:39:00.040 INFO:tasks.ceph.mgr.x.smithi111.stderr:2023-10-03T17:39:00.041+0000 7fe483b87640 -1 librbd::ImageState: 0x55f87e868e80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:39:00.042 INFO:tasks.ceph.mgr.x.smithi111.stderr:2023-10-03T17:39:00.045+0000 7fe483b87640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:39:00.042 INFO:tasks.ceph.mgr.x.smithi111.stderr:2023-10-03T17:39:00.045+0000 7fe483b87640 -1 librbd::ImageState: 0x55f87e869a80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:39:00.046 INFO:tasks.ceph.mgr.x.smithi111.stderr:2023-10-03T17:39:00.049+0000 7fe483b87640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:39:00.046 INFO:tasks.ceph.mgr.x.smithi111.stderr:2023-10-03T17:39:00.049+0000 7fe483b87640 -1 librbd::ImageState: 0x55f87e9cc000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:39:00.049 INFO:tasks.ceph.mgr.x.smithi111.stderr:2023-10-03T17:39:00.049+0000 7fe483b87640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:39:00.049 INFO:tasks.ceph.mgr.x.smithi111.stderr:2023-10-03T17:39:00.049+0000 7fe483b87640 -1 librbd::ImageState: 0x55f87e9cc200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:39:00.054 INFO:tasks.ceph.mgr.x.smithi111.stderr:2023-10-03T17:39:00.053+0000 7fe483b87640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:39:00.054 INFO:tasks.ceph.mgr.x.smithi111.stderr:2023-10-03T17:39:00.053+0000 7fe483b87640 -1 librbd::ImageState: 0x55f87e9cc400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:39:00.055 INFO:tasks.ceph.mgr.x.smithi111.stderr:2023-10-03T17:39:00.057+0000 7fe483b87640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:39:00.055 INFO:tasks.ceph.mgr.x.smithi111.stderr:2023-10-03T17:39:00.057+0000 7fe483b87640 -1 librbd::ImageState: 0x55f87e9cc600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:39:00.059 INFO:tasks.ceph.mgr.x.smithi111.stderr:2023-10-03T17:39:00.061+0000 7fe483b87640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:39:00.059 INFO:tasks.ceph.mgr.x.smithi111.stderr:2023-10-03T17:39:00.061+0000 7fe483b87640 -1 librbd::ImageState: 0x55f87e9cc800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:39:00.065 INFO:tasks.ceph.mgr.x.smithi111.stderr:2023-10-03T17:39:00.065+0000 7fe483b87640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:39:00.065 INFO:tasks.ceph.mgr.x.smithi111.stderr:2023-10-03T17:39:00.065+0000 7fe483b87640 -1 librbd::ImageState: 0x55f87e9cca00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:39:00.071 INFO:tasks.ceph.mgr.x.smithi111.stderr:2023-10-03T17:39:00.073+0000 7fe483b87640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:39:00.071 INFO:tasks.ceph.mgr.x.smithi111.stderr:2023-10-03T17:39:00.073+0000 7fe483b87640 -1 librbd::ImageState: 0x55f87e9ccc00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:39:00.077 INFO:tasks.ceph.mgr.x.smithi111.stderr:2023-10-03T17:39:00.077+0000 7fe483b87640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:39:00.077 INFO:tasks.ceph.mgr.x.smithi111.stderr:2023-10-03T17:39:00.077+0000 7fe483b87640 -1 librbd::ImageState: 0x55f87e9cce00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:39:01.999 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:39:01.997+0000 7f09e4851640 -1 reset not still connected to 0x557d76621040 2023-10-03T17:39:01.999 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:39:01.997+0000 7f09e4851640 -1 reset not still connected to 0x557d7664e750 2023-10-03T17:39:07.017 INFO:tasks.workunit.client.0.smithi111.stderr:++ ceph mgr dump 2023-10-03T17:39:07.018 INFO:tasks.workunit.client.0.smithi111.stderr:++ jq '.active_clients[]' 2023-10-03T17:39:07.018 INFO:tasks.workunit.client.0.smithi111.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T17:39:07.043 INFO:tasks.workunit.client.0.smithi111.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T17:39:07.959 INFO:tasks.workunit.client.0.smithi111.stderr:+ CLIENT_ADDR=172.21.15.111:0/4228986752 2023-10-03T17:39:07.961 INFO:tasks.workunit.client.0.smithi111.stderr:++ date +%s 2023-10-03T17:39:07.962 INFO:tasks.workunit.client.0.smithi111.stderr:+ CURRENT_TIME=1696354747 2023-10-03T17:39:07.962 INFO:tasks.workunit.client.0.smithi111.stderr:+ [[ 1696354747 -le 1696358185 ]] 2023-10-03T17:39:07.963 INFO:tasks.workunit.client.0.smithi111.stderr:+ [[ 172.21.15.111:0/4228986752 != \1\7\2\.\2\1\.\1\5\.\1\1\1\:\0\/\4\2\2\8\9\8\6\7\5\2 ]] 2023-10-03T17:39:07.963 INFO:tasks.workunit.client.0.smithi111.stderr:+ sleep 10 2023-10-03T17:39:17.970 INFO:tasks.workunit.client.0.smithi111.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T17:39:17.971 INFO:tasks.workunit.client.0.smithi111.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T17:39:17.974 INFO:tasks.workunit.client.0.smithi111.stderr:++ jq '.active_clients[]' 2023-10-03T17:39:17.978 INFO:tasks.workunit.client.0.smithi111.stderr:++ ceph mgr dump 2023-10-03T17:39:18.922 INFO:tasks.workunit.client.0.smithi111.stderr:+ CLIENT_ADDR=172.21.15.111:0/4228986752 2023-10-03T17:39:18.922 INFO:tasks.workunit.client.0.smithi111.stderr:++ date +%s 2023-10-03T17:39:18.923 INFO:tasks.workunit.client.0.smithi111.stderr:+ CURRENT_TIME=1696354758 2023-10-03T17:39:18.923 INFO:tasks.workunit.client.0.smithi111.stderr:+ [[ 1696354758 -le 1696358185 ]] 2023-10-03T17:39:18.923 INFO:tasks.workunit.client.0.smithi111.stderr:+ [[ 172.21.15.111:0/4228986752 != \1\7\2\.\2\1\.\1\5\.\1\1\1\:\0\/\4\2\2\8\9\8\6\7\5\2 ]] 2023-10-03T17:39:18.923 INFO:tasks.workunit.client.0.smithi111.stderr:+ sleep 10 2023-10-03T17:39:28.926 INFO:tasks.workunit.client.0.smithi111.stderr:++ jq '.active_clients[]' 2023-10-03T17:39:28.926 INFO:tasks.workunit.client.0.smithi111.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T17:39:28.929 INFO:tasks.workunit.client.0.smithi111.stderr:++ ceph mgr dump 2023-10-03T17:39:28.930 INFO:tasks.workunit.client.0.smithi111.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T17:39:29.838 INFO:tasks.workunit.client.0.smithi111.stderr:+ CLIENT_ADDR=172.21.15.111:0/4228986752 2023-10-03T17:39:29.838 INFO:tasks.workunit.client.0.smithi111.stderr:++ date +%s 2023-10-03T17:39:29.839 INFO:tasks.workunit.client.0.smithi111.stderr:+ CURRENT_TIME=1696354769 2023-10-03T17:39:29.839 INFO:tasks.workunit.client.0.smithi111.stderr:+ [[ 1696354769 -le 1696358185 ]] 2023-10-03T17:39:29.839 INFO:tasks.workunit.client.0.smithi111.stderr:+ [[ 172.21.15.111:0/4228986752 != \1\7\2\.\2\1\.\1\5\.\1\1\1\:\0\/\4\2\2\8\9\8\6\7\5\2 ]] 2023-10-03T17:39:29.840 INFO:tasks.workunit.client.0.smithi111.stderr:+ sleep 10 2023-10-03T17:39:39.848 INFO:tasks.workunit.client.0.smithi111.stderr:++ ceph mgr dump 2023-10-03T17:39:39.848 INFO:tasks.workunit.client.0.smithi111.stderr:++ jq '.active_clients[]' 2023-10-03T17:39:39.849 INFO:tasks.workunit.client.0.smithi111.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T17:39:39.857 INFO:tasks.workunit.client.0.smithi111.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T17:39:40.679 INFO:tasks.workunit.client.0.smithi111.stderr:+ CLIENT_ADDR=172.21.15.111:0/4228986752 2023-10-03T17:39:40.680 INFO:tasks.workunit.client.0.smithi111.stderr:++ date +%s 2023-10-03T17:39:40.680 INFO:tasks.workunit.client.0.smithi111.stderr:+ CURRENT_TIME=1696354780 2023-10-03T17:39:40.680 INFO:tasks.workunit.client.0.smithi111.stderr:+ [[ 1696354780 -le 1696358185 ]] 2023-10-03T17:39:40.681 INFO:tasks.workunit.client.0.smithi111.stderr:+ [[ 172.21.15.111:0/4228986752 != \1\7\2\.\2\1\.\1\5\.\1\1\1\:\0\/\4\2\2\8\9\8\6\7\5\2 ]] 2023-10-03T17:39:40.681 INFO:tasks.workunit.client.0.smithi111.stderr:+ sleep 10 2023-10-03T17:39:50.239 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:39:50.240+0000 7f09e4851640 -1 reset not still connected to 0x557d75f2b2b0 2023-10-03T17:39:50.239 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:39:50.240+0000 7f09e4851640 -1 reset not still connected to 0x557d76621040 2023-10-03T17:39:50.240 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:39:50.240+0000 7f09e4851640 -1 reset not still connected to 0x557d7664e750 2023-10-03T17:39:50.693 INFO:tasks.workunit.client.0.smithi111.stderr:++ ceph mgr dump 2023-10-03T17:39:50.693 INFO:tasks.workunit.client.0.smithi111.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T17:39:50.697 INFO:tasks.workunit.client.0.smithi111.stderr:++ jq '.active_clients[]' 2023-10-03T17:39:50.712 INFO:tasks.workunit.client.0.smithi111.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T17:39:51.659 INFO:tasks.workunit.client.0.smithi111.stderr:+ CLIENT_ADDR=172.21.15.111:0/4228986752 2023-10-03T17:39:51.659 INFO:tasks.workunit.client.0.smithi111.stderr:++ date +%s 2023-10-03T17:39:51.659 INFO:tasks.workunit.client.0.smithi111.stderr:+ CURRENT_TIME=1696354791 2023-10-03T17:39:51.659 INFO:tasks.workunit.client.0.smithi111.stderr:+ [[ 1696354791 -le 1696358185 ]] 2023-10-03T17:39:51.660 INFO:tasks.workunit.client.0.smithi111.stderr:+ [[ 172.21.15.111:0/4228986752 != \1\7\2\.\2\1\.\1\5\.\1\1\1\:\0\/\4\2\2\8\9\8\6\7\5\2 ]] 2023-10-03T17:39:51.660 INFO:tasks.workunit.client.0.smithi111.stderr:+ sleep 10 2023-10-03T17:40:01.664 INFO:tasks.workunit.client.0.smithi111.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T17:40:01.665 INFO:tasks.workunit.client.0.smithi111.stderr:++ jq '.active_clients[]' 2023-10-03T17:40:01.665 INFO:tasks.workunit.client.0.smithi111.stderr:++ ceph mgr dump 2023-10-03T17:40:01.690 INFO:tasks.workunit.client.0.smithi111.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T17:40:02.601 INFO:tasks.workunit.client.0.smithi111.stderr:+ CLIENT_ADDR= 2023-10-03T17:40:02.601 INFO:tasks.workunit.client.0.smithi111.stderr:++ date +%s 2023-10-03T17:40:02.601 INFO:tasks.workunit.client.0.smithi111.stderr:+ CURRENT_TIME=1696354802 2023-10-03T17:40:02.601 INFO:tasks.workunit.client.0.smithi111.stderr:+ [[ 1696354802 -le 1696358185 ]] 2023-10-03T17:40:02.602 INFO:tasks.workunit.client.0.smithi111.stderr:+ [[ '' != \1\7\2\.\2\1\.\1\5\.\1\1\1\:\0\/\4\2\2\8\9\8\6\7\5\2 ]] 2023-10-03T17:40:02.602 INFO:tasks.workunit.client.0.smithi111.stderr:+ ceph osd blocklist add 2023-10-03T17:40:02.966 INFO:tasks.workunit.client.0.smithi111.stderr:2023-10-03T17:40:02.968+0000 7f5f21a7b640 0 --2- 172.21.15.111:0/3826111065 >> v2:172.21.15.111:3300/0 conn(0x7f5f1401e0d0 0x7f5f1401e4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2023-10-03T17:40:03.371 INFO:tasks.workunit.client.0.smithi111.stderr:Invalid command: missing required parameter addr() 2023-10-03T17:40:03.371 INFO:tasks.workunit.client.0.smithi111.stderr:osd blocklist [] [] : add (optionally until seconds from now) or remove from blocklist 2023-10-03T17:40:03.372 INFO:tasks.workunit.client.0.smithi111.stderr:Error EINVAL: invalid command 2023-10-03T17:40:03.375 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-10-03T17:40:03.375 INFO:tasks.workunit:Stopping ['rbd/rbd_support_module_recovery.sh'] on client.0... 2023-10-03T17:40:03.375 DEBUG:teuthology.orchestra.run.smithi111:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2023-10-03T17:40:03.858 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_a7de91bfa5778a0d559a7e9bc8e35f93456b41ad/qa/tasks/workunit.py", line 129, in task p.spawn(_run_tests, ctx, refspec, role, tests, File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_a7de91bfa5778a0d559a7e9bc8e35f93456b41ad/qa/tasks/workunit.py", line 424, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test rbd/rbd_support_module_recovery.sh) on smithi111 with status 22: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a7de91bfa5778a0d559a7e9bc8e35f93456b41ad TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/rbd_support_module_recovery.sh' 2023-10-03T17:40:03.924 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=0d3bf0ae55a041efb7d9f3ffc22fb248 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_a7de91bfa5778a0d559a7e9bc8e35f93456b41ad/qa/tasks/workunit.py", line 129, in task p.spawn(_run_tests, ctx, refspec, role, tests, File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_a7de91bfa5778a0d559a7e9bc8e35f93456b41ad/qa/tasks/workunit.py", line 424, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test rbd/rbd_support_module_recovery.sh) on smithi111 with status 22: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a7de91bfa5778a0d559a7e9bc8e35f93456b41ad TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/rbd_support_module_recovery.sh' 2023-10-03T17:40:03.927 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-10-03T17:40:03.983 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-10-03T17:40:03.994 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-10-03T17:40:03.994 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T17:40:04.609 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-03T17:40:04.611 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-10-03T17:40:04.638 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":208,"stamp":"2023-10-03T17:40:04.025047+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20635416419,"num_objects":5186,"num_object_clones":0,"num_object_copies":10372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5186,"num_whiteouts":0,"num_read":131900,"num_read_kb":451016,"num_write":130966,"num_write_kb":524595,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43867,"ondisk_log_size":43867,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1864324,"kb_used_data":1045036,"kb_used_omap":4,"kb_used_meta":819259,"kb_avail":281251196,"statfs":{"total":289910292480,"available":288001224704,"internally_reserved":0,"allocated":1070116864,"data_stored":1070158750,"data_compressed":7460,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4782,"internal_metadata":838921554},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,3,0,6,9,15,9],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":65814528,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8885,"num_read_kb":31647,"num_write":7614,"num_write_kb":30456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006264"},"pg_stats":[{"pgid":"2.7","version":"46'632","reported_seq":1090,"reported_epoch":46,"state":"active+clean","last_fresh":"2023-10-03T17:40:01.867643+0000","last_change":"2023-10-03T17:34:42.277381+0000","last_active":"2023-10-03T17:40:01.867643+0000","last_peered":"2023-10-03T17:40:01.867643+0000","last_clean":"2023-10-03T17:40:01.867643+0000","last_became_active":"2023-10-03T17:34:42.276624+0000","last_became_peered":"2023-10-03T17:34:42.276624+0000","last_unstale":"2023-10-03T17:40:01.867643+0000","last_undegraded":"2023-10-03T17:40:01.867643+0000","last_fullsized":"2023-10-03T17:40:01.867643+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:34:41.238033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:34:41.238033+0000","last_clean_scrub_stamp":"2023-10-03T17:34:41.238033+0000","objects_scrubbed":0,"log_size":632,"log_dups_size":0,"ondisk_log_size":632,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T18:56:53.056366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":132,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":837,"num_read_kb":699,"num_write":16,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"46'1077","reported_seq":2044,"reported_epoch":46,"state":"active+clean","last_fresh":"2023-10-03T17:40:01.867049+0000","last_change":"2023-10-03T17:34:42.271628+0000","last_active":"2023-10-03T17:40:01.867049+0000","last_peered":"2023-10-03T17:40:01.867049+0000","last_clean":"2023-10-03T17:40:01.867049+0000","last_became_active":"2023-10-03T17:34:42.271201+0000","last_became_peered":"2023-10-03T17:34:42.271201+0000","last_unstale":"2023-10-03T17:40:01.867049+0000","last_undegraded":"2023-10-03T17:40:01.867049+0000","last_fullsized":"2023-10-03T17:40:01.867049+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:34:41.238033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:34:41.238033+0000","last_clean_scrub_stamp":"2023-10-03T17:34:41.238033+0000","objects_scrubbed":0,"log_size":1077,"log_dups_size":0,"ondisk_log_size":1077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:13:49.428992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":218,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1806,"num_read_kb":1593,"num_write":68,"num_write_kb":68,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"39'7","reported_seq":72,"reported_epoch":45,"state":"active+clean","last_fresh":"2023-10-03T17:38:55.232572+0000","last_change":"2023-10-03T17:34:42.277773+0000","last_active":"2023-10-03T17:38:55.232572+0000","last_peered":"2023-10-03T17:38:55.232572+0000","last_clean":"2023-10-03T17:38:55.232572+0000","last_became_active":"2023-10-03T17:34:42.276941+0000","last_became_peered":"2023-10-03T17:34:42.276941+0000","last_unstale":"2023-10-03T17:38:55.232572+0000","last_undegraded":"2023-10-03T17:38:55.232572+0000","last_fullsized":"2023-10-03T17:38:55.232572+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:34:41.238033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:34:41.238033+0000","last_clean_scrub_stamp":"2023-10-03T17:34:41.238033+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T21:07:05.154169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":249,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"46'865","reported_seq":1605,"reported_epoch":46,"state":"active+clean","last_fresh":"2023-10-03T17:40:01.869122+0000","last_change":"2023-10-03T17:34:42.276876+0000","last_active":"2023-10-03T17:40:01.869122+0000","last_peered":"2023-10-03T17:40:01.869122+0000","last_clean":"2023-10-03T17:40:01.869122+0000","last_became_active":"2023-10-03T17:34:42.276459+0000","last_became_peered":"2023-10-03T17:34:42.276459+0000","last_unstale":"2023-10-03T17:40:01.869122+0000","last_undegraded":"2023-10-03T17:40:01.869122+0000","last_fullsized":"2023-10-03T17:40:01.869122+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:34:41.238033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:34:41.238033+0000","last_clean_scrub_stamp":"2023-10-03T17:34:41.238033+0000","objects_scrubbed":0,"log_size":865,"log_dups_size":0,"ondisk_log_size":865,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:03:40.525005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":80,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":1294,"num_read_kb":1085,"num_write":41,"num_write_kb":41,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"46'33692","reported_seq":66023,"reported_epoch":46,"state":"active+clean","last_fresh":"2023-10-03T17:40:03.016087+0000","last_change":"2023-10-03T17:34:50.396952+0000","last_active":"2023-10-03T17:40:03.016087+0000","last_peered":"2023-10-03T17:40:03.016087+0000","last_clean":"2023-10-03T17:40:03.016087+0000","last_became_active":"2023-10-03T17:34:48.310701+0000","last_became_peered":"2023-10-03T17:34:48.310701+0000","last_unstale":"2023-10-03T17:40:03.016087+0000","last_undegraded":"2023-10-03T17:40:03.016087+0000","last_fullsized":"2023-10-03T17:40:03.016087+0000","mapping_epoch":16,"log_start":"44'23600","ondisk_log_start":"44'23600","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_clean_scrub_stamp":"2023-10-03T17:34:47.281095+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T21:26:47.697148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040768800000000002,"stat_sum":{"num_bytes":5317832704,"num_objects":1320,"num_object_clones":0,"num_object_copies":2640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1320,"num_whiteouts":0,"num_read":32269,"num_read_kb":114528,"num_write":33692,"num_write_kb":134768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"46'822","reported_seq":1427,"reported_epoch":46,"state":"active+clean","last_fresh":"2023-10-03T17:40:02.605566+0000","last_change":"2023-10-03T17:34:42.271479+0000","last_active":"2023-10-03T17:40:02.605566+0000","last_peered":"2023-10-03T17:40:02.605566+0000","last_clean":"2023-10-03T17:40:02.605566+0000","last_became_active":"2023-10-03T17:34:42.270625+0000","last_became_peered":"2023-10-03T17:34:42.270625+0000","last_unstale":"2023-10-03T17:40:02.605566+0000","last_undegraded":"2023-10-03T17:40:02.605566+0000","last_fullsized":"2023-10-03T17:40:02.605566+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:34:41.238033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:34:41.238033+0000","last_clean_scrub_stamp":"2023-10-03T17:34:41.238033+0000","objects_scrubbed":0,"log_size":822,"log_dups_size":0,"ondisk_log_size":822,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:29:20.305630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":200,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1127,"num_read_kb":944,"num_write":42,"num_write_kb":42,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"46'31291","reported_seq":61917,"reported_epoch":46,"state":"active+clean","last_fresh":"2023-10-03T17:40:03.016378+0000","last_change":"2023-10-03T17:34:50.321236+0000","last_active":"2023-10-03T17:40:03.016378+0000","last_peered":"2023-10-03T17:40:03.016378+0000","last_clean":"2023-10-03T17:40:03.016378+0000","last_became_active":"2023-10-03T17:34:48.310555+0000","last_became_peered":"2023-10-03T17:34:48.310555+0000","last_unstale":"2023-10-03T17:40:03.016378+0000","last_undegraded":"2023-10-03T17:40:03.016378+0000","last_fullsized":"2023-10-03T17:40:03.016378+0000","mapping_epoch":16,"log_start":"44'21200","ondisk_log_start":"44'21200","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_clean_scrub_stamp":"2023-10-03T17:34:47.281095+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T19:53:40.901598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086289699999999995,"stat_sum":{"num_bytes":5000892416,"num_objects":1242,"num_object_clones":0,"num_object_copies":2484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1242,"num_whiteouts":0,"num_read":30561,"num_read_kb":107880,"num_write":31291,"num_write_kb":125164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"46'24","reported_seq":471,"reported_epoch":46,"state":"active+clean","last_fresh":"2023-10-03T17:40:01.861419+0000","last_change":"2023-10-03T17:34:42.278925+0000","last_active":"2023-10-03T17:40:01.861419+0000","last_peered":"2023-10-03T17:40:01.861419+0000","last_clean":"2023-10-03T17:40:01.861419+0000","last_became_active":"2023-10-03T17:34:42.278196+0000","last_became_peered":"2023-10-03T17:34:42.278196+0000","last_unstale":"2023-10-03T17:40:01.861419+0000","last_undegraded":"2023-10-03T17:40:01.861419+0000","last_fullsized":"2023-10-03T17:40:01.861419+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:34:41.238033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:34:41.238033+0000","last_clean_scrub_stamp":"2023-10-03T17:34:41.238033+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T00:36:18.073012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":823,"num_read_kb":694,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"46'32385","reported_seq":63458,"reported_epoch":46,"state":"active+clean","last_fresh":"2023-10-03T17:40:02.995867+0000","last_change":"2023-10-03T17:34:50.396806+0000","last_active":"2023-10-03T17:40:02.995867+0000","last_peered":"2023-10-03T17:40:02.995867+0000","last_clean":"2023-10-03T17:40:02.995867+0000","last_became_active":"2023-10-03T17:34:48.310912+0000","last_became_peered":"2023-10-03T17:34:48.310912+0000","last_unstale":"2023-10-03T17:40:02.995867+0000","last_undegraded":"2023-10-03T17:40:02.995867+0000","last_fullsized":"2023-10-03T17:40:02.995867+0000","mapping_epoch":16,"log_start":"44'22300","ondisk_log_start":"44'22300","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_clean_scrub_stamp":"2023-10-03T17:34:47.281095+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T18:44:12.843757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045661900000000001,"stat_sum":{"num_bytes":5124481024,"num_objects":1271,"num_object_clones":0,"num_object_copies":2542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1271,"num_whiteouts":0,"num_read":31006,"num_read_kb":110640,"num_write":32385,"num_write_kb":129540,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"46'13","reported_seq":211,"reported_epoch":46,"state":"active+clean","last_fresh":"2023-10-03T17:40:01.840287+0000","last_change":"2023-10-03T17:34:42.273043+0000","last_active":"2023-10-03T17:40:01.840287+0000","last_peered":"2023-10-03T17:40:01.840287+0000","last_clean":"2023-10-03T17:40:01.840287+0000","last_became_active":"2023-10-03T17:34:42.272306+0000","last_became_peered":"2023-10-03T17:34:42.272306+0000","last_unstale":"2023-10-03T17:40:01.840287+0000","last_undegraded":"2023-10-03T17:40:01.840287+0000","last_fullsized":"2023-10-03T17:40:01.840287+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:34:41.238033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:34:41.238033+0000","last_clean_scrub_stamp":"2023-10-03T17:34:41.238033+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T18:02:59.747348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":200,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":309,"num_read_kb":242,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"46'33258","reported_seq":65018,"reported_epoch":46,"state":"active+clean","last_fresh":"2023-10-03T17:40:03.017249+0000","last_change":"2023-10-03T17:34:50.322676+0000","last_active":"2023-10-03T17:40:03.017249+0000","last_peered":"2023-10-03T17:40:03.017249+0000","last_clean":"2023-10-03T17:40:03.017249+0000","last_became_active":"2023-10-03T17:34:48.310612+0000","last_became_peered":"2023-10-03T17:34:48.310612+0000","last_unstale":"2023-10-03T17:40:03.017249+0000","last_undegraded":"2023-10-03T17:40:03.017249+0000","last_fullsized":"2023-10-03T17:40:03.017249+0000","mapping_epoch":16,"log_start":"44'23200","ondisk_log_start":"44'23200","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_clean_scrub_stamp":"2023-10-03T17:34:47.281095+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T20:53:44.523856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022375120000000001,"stat_sum":{"num_bytes":5191618579,"num_objects":1288,"num_object_clones":0,"num_object_copies":2576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1288,"num_whiteouts":0,"num_read":31697,"num_read_kb":112600,"num_write":33258,"num_write_kb":133026,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"36'4","reported_seq":62,"reported_epoch":45,"state":"active+clean","last_fresh":"2023-10-03T17:38:55.232552+0000","last_change":"2023-10-03T17:34:42.271102+0000","last_active":"2023-10-03T17:38:55.232552+0000","last_peered":"2023-10-03T17:38:55.232552+0000","last_clean":"2023-10-03T17:38:55.232552+0000","last_became_active":"2023-10-03T17:34:42.270194+0000","last_became_peered":"2023-10-03T17:34:42.270194+0000","last_unstale":"2023-10-03T17:38:55.232552+0000","last_undegraded":"2023-10-03T17:38:55.232552+0000","last_fullsized":"2023-10-03T17:38:55.232552+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:34:41.238033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:34:41.238033+0000","last_clean_scrub_stamp":"2023-10-03T17:34:41.238033+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:41:26.441632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":132,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":8,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":174,"reported_epoch":45,"state":"active+clean","last_fresh":"2023-10-03T17:38:55.237074+0000","last_change":"2023-10-03T17:34:39.504748+0000","last_active":"2023-10-03T17:38:55.237074+0000","last_peered":"2023-10-03T17:38:55.237074+0000","last_clean":"2023-10-03T17:38:55.237074+0000","last_became_active":"2023-10-03T17:34:39.502393+0000","last_became_peered":"2023-10-03T17:34:39.502393+0000","last_unstale":"2023-10-03T17:38:55.237074+0000","last_undegraded":"2023-10-03T17:38:55.237074+0000","last_fullsized":"2023-10-03T17:38:55.237074+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:34:38.227481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:34:38.227481+0000","last_clean_scrub_stamp":"2023-10-03T17:34:38.227481+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T00:10:27.940656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":20634824723,"num_objects":5121,"num_object_clones":0,"num_object_copies":10242,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5121,"num_whiteouts":0,"num_read":125533,"num_read_kb":445648,"num_write":130626,"num_write_kb":522498,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1070002176,"data_stored":1069994022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40326,"ondisk_log_size":40326,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":1328,"num_objects":63,"num_object_clones":0,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63,"num_whiteouts":0,"num_read":6261,"num_read_kb":5277,"num_write":203,"num_write_kb":203,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":2656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3444,"ondisk_log_size":3444,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":7460,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738439,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":932344,"kb_used_data":522352,"kb_used_omap":1,"kb_used_meta":409982,"kb_avail":93439496,"statfs":{"total":96636764160,"available":95682043904,"internally_reserved":0,"allocated":534888448,"data_stored":534601244,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":419822022},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,3,2,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 17:39:44 2023","interfaces":[{"interface":"back","average":{"1min":0.375,"5min":0.45000000000000001,"15min":0.49399999999999999},"min":{"1min":0.182,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":1.982,"5min":4.4640000000000004,"15min":4.4640000000000004},"last":0.25800000000000001},{"interface":"front","average":{"1min":0.42499999999999999,"5min":0.41699999999999998,"15min":0.51000000000000001},"min":{"1min":0.19800000000000001,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":1.897,"5min":1.897,"15min":1.897},"last":0.27900000000000003}]},{"osd":1,"last update":"Tue Oct 3 17:39:44 2023","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.46700000000000003,"15min":0.52600000000000002},"min":{"1min":0.20300000000000001,"5min":0.16600000000000001,"15min":0.16600000000000001},"max":{"1min":2.2040000000000002,"5min":4.4260000000000002,"15min":4.4260000000000002},"last":0.22},{"interface":"front","average":{"1min":0.375,"5min":0.41699999999999998,"15min":0.54000000000000004},"min":{"1min":0.223,"5min":0.192,"15min":0.192},"max":{"1min":0.70499999999999996,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.23599999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738438,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":469716,"kb_used_data":258124,"kb_used_omap":1,"kb_used_meta":211582,"kb_avail":93902124,"statfs":{"total":96636764160,"available":96155774976,"internally_reserved":0,"allocated":264318976,"data_stored":264455083,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":216660422},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,0,1,4,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 17:39:56 2023","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.39200000000000002,"15min":0.47799999999999998},"min":{"1min":0.187,"5min":0.161,"15min":0.161},"max":{"1min":1.577,"5min":4.1879999999999997,"15min":4.1879999999999997},"last":0.218},{"interface":"front","average":{"1min":0.371,"5min":0.38700000000000001,"15min":0.48799999999999999},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":1.3899999999999999,"5min":1.532,"15min":1.532},"last":0.19900000000000001}]},{"osd":2,"last update":"Tue Oct 3 17:39:56 2023","interfaces":[{"interface":"back","average":{"1min":0.37,"5min":0.38700000000000001,"15min":0.49099999999999999},"min":{"1min":0.17799999999999999,"5min":0.159,"15min":0.159},"max":{"1min":2.4889999999999999,"5min":2.4889999999999999,"15min":2.4889999999999999},"last":3.0390000000000001},{"interface":"front","average":{"1min":0.39600000000000002,"5min":0.39700000000000002,"15min":0.51900000000000002},"min":{"1min":0.19400000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":2.5219999999999998,"5min":2.5219999999999998,"15min":2.5219999999999998},"last":3.0699999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738438,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":462264,"kb_used_data":264560,"kb_used_omap":1,"kb_used_meta":197694,"kb_avail":93909576,"statfs":{"total":96636764160,"available":96163405824,"internally_reserved":0,"allocated":270909440,"data_stored":271102423,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":202439110},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,2,3,10,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 17:39:43 2023","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.38400000000000001,"15min":0.42399999999999999},"min":{"1min":0.17299999999999999,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":3.2269999999999999,"5min":3.2269999999999999,"15min":3.2269999999999999},"last":0.19700000000000001},{"interface":"front","average":{"1min":0.28499999999999998,"5min":0.41499999999999998,"15min":0.45800000000000002},"min":{"1min":0.19,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.45800000000000002,"5min":2.1680000000000001,"15min":2.1680000000000001},"last":0.22600000000000001}]},{"osd":2,"last update":"Tue Oct 3 17:39:43 2023","interfaces":[{"interface":"back","average":{"1min":0.32300000000000001,"5min":0.46300000000000002,"15min":0.54300000000000004},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.61399999999999999,"5min":2.823,"15min":2.823},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.47099999999999997,"15min":0.52900000000000003},"min":{"1min":0.249,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":2.9239999999999999,"5min":2.9239999999999999,"15min":2.9239999999999999},"last":0.26100000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":879,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":270651392,"data_stored":270651392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":264388608,"data_stored":264384531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":534962176,"data_stored":534958099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-03T17:40:04.639 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T17:40:05.133 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-03T17:40:05.134 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-10-03T17:40:05.168 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":208,"stamp":"2023-10-03T17:40:04.025047+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20635416419,"num_objects":5186,"num_object_clones":0,"num_object_copies":10372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5186,"num_whiteouts":0,"num_read":131900,"num_read_kb":451016,"num_write":130966,"num_write_kb":524595,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43867,"ondisk_log_size":43867,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1864324,"kb_used_data":1045036,"kb_used_omap":4,"kb_used_meta":819259,"kb_avail":281251196,"statfs":{"total":289910292480,"available":288001224704,"internally_reserved":0,"allocated":1070116864,"data_stored":1070158750,"data_compressed":7460,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4782,"internal_metadata":838921554},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,3,0,6,9,15,9],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":65814528,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8885,"num_read_kb":31647,"num_write":7614,"num_write_kb":30456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006264"},"pg_stats":[{"pgid":"2.7","version":"46'632","reported_seq":1090,"reported_epoch":46,"state":"active+clean","last_fresh":"2023-10-03T17:40:01.867643+0000","last_change":"2023-10-03T17:34:42.277381+0000","last_active":"2023-10-03T17:40:01.867643+0000","last_peered":"2023-10-03T17:40:01.867643+0000","last_clean":"2023-10-03T17:40:01.867643+0000","last_became_active":"2023-10-03T17:34:42.276624+0000","last_became_peered":"2023-10-03T17:34:42.276624+0000","last_unstale":"2023-10-03T17:40:01.867643+0000","last_undegraded":"2023-10-03T17:40:01.867643+0000","last_fullsized":"2023-10-03T17:40:01.867643+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:34:41.238033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:34:41.238033+0000","last_clean_scrub_stamp":"2023-10-03T17:34:41.238033+0000","objects_scrubbed":0,"log_size":632,"log_dups_size":0,"ondisk_log_size":632,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T18:56:53.056366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":132,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":837,"num_read_kb":699,"num_write":16,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"46'1077","reported_seq":2044,"reported_epoch":46,"state":"active+clean","last_fresh":"2023-10-03T17:40:01.867049+0000","last_change":"2023-10-03T17:34:42.271628+0000","last_active":"2023-10-03T17:40:01.867049+0000","last_peered":"2023-10-03T17:40:01.867049+0000","last_clean":"2023-10-03T17:40:01.867049+0000","last_became_active":"2023-10-03T17:34:42.271201+0000","last_became_peered":"2023-10-03T17:34:42.271201+0000","last_unstale":"2023-10-03T17:40:01.867049+0000","last_undegraded":"2023-10-03T17:40:01.867049+0000","last_fullsized":"2023-10-03T17:40:01.867049+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:34:41.238033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:34:41.238033+0000","last_clean_scrub_stamp":"2023-10-03T17:34:41.238033+0000","objects_scrubbed":0,"log_size":1077,"log_dups_size":0,"ondisk_log_size":1077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:13:49.428992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":218,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1806,"num_read_kb":1593,"num_write":68,"num_write_kb":68,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"39'7","reported_seq":72,"reported_epoch":45,"state":"active+clean","last_fresh":"2023-10-03T17:38:55.232572+0000","last_change":"2023-10-03T17:34:42.277773+0000","last_active":"2023-10-03T17:38:55.232572+0000","last_peered":"2023-10-03T17:38:55.232572+0000","last_clean":"2023-10-03T17:38:55.232572+0000","last_became_active":"2023-10-03T17:34:42.276941+0000","last_became_peered":"2023-10-03T17:34:42.276941+0000","last_unstale":"2023-10-03T17:38:55.232572+0000","last_undegraded":"2023-10-03T17:38:55.232572+0000","last_fullsized":"2023-10-03T17:38:55.232572+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:34:41.238033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:34:41.238033+0000","last_clean_scrub_stamp":"2023-10-03T17:34:41.238033+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T21:07:05.154169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":249,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"46'865","reported_seq":1605,"reported_epoch":46,"state":"active+clean","last_fresh":"2023-10-03T17:40:01.869122+0000","last_change":"2023-10-03T17:34:42.276876+0000","last_active":"2023-10-03T17:40:01.869122+0000","last_peered":"2023-10-03T17:40:01.869122+0000","last_clean":"2023-10-03T17:40:01.869122+0000","last_became_active":"2023-10-03T17:34:42.276459+0000","last_became_peered":"2023-10-03T17:34:42.276459+0000","last_unstale":"2023-10-03T17:40:01.869122+0000","last_undegraded":"2023-10-03T17:40:01.869122+0000","last_fullsized":"2023-10-03T17:40:01.869122+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:34:41.238033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:34:41.238033+0000","last_clean_scrub_stamp":"2023-10-03T17:34:41.238033+0000","objects_scrubbed":0,"log_size":865,"log_dups_size":0,"ondisk_log_size":865,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:03:40.525005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":80,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":1294,"num_read_kb":1085,"num_write":41,"num_write_kb":41,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"46'33692","reported_seq":66023,"reported_epoch":46,"state":"active+clean","last_fresh":"2023-10-03T17:40:03.016087+0000","last_change":"2023-10-03T17:34:50.396952+0000","last_active":"2023-10-03T17:40:03.016087+0000","last_peered":"2023-10-03T17:40:03.016087+0000","last_clean":"2023-10-03T17:40:03.016087+0000","last_became_active":"2023-10-03T17:34:48.310701+0000","last_became_peered":"2023-10-03T17:34:48.310701+0000","last_unstale":"2023-10-03T17:40:03.016087+0000","last_undegraded":"2023-10-03T17:40:03.016087+0000","last_fullsized":"2023-10-03T17:40:03.016087+0000","mapping_epoch":16,"log_start":"44'23600","ondisk_log_start":"44'23600","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_clean_scrub_stamp":"2023-10-03T17:34:47.281095+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T21:26:47.697148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040768800000000002,"stat_sum":{"num_bytes":5317832704,"num_objects":1320,"num_object_clones":0,"num_object_copies":2640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1320,"num_whiteouts":0,"num_read":32269,"num_read_kb":114528,"num_write":33692,"num_write_kb":134768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"46'822","reported_seq":1427,"reported_epoch":46,"state":"active+clean","last_fresh":"2023-10-03T17:40:02.605566+0000","last_change":"2023-10-03T17:34:42.271479+0000","last_active":"2023-10-03T17:40:02.605566+0000","last_peered":"2023-10-03T17:40:02.605566+0000","last_clean":"2023-10-03T17:40:02.605566+0000","last_became_active":"2023-10-03T17:34:42.270625+0000","last_became_peered":"2023-10-03T17:34:42.270625+0000","last_unstale":"2023-10-03T17:40:02.605566+0000","last_undegraded":"2023-10-03T17:40:02.605566+0000","last_fullsized":"2023-10-03T17:40:02.605566+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:34:41.238033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:34:41.238033+0000","last_clean_scrub_stamp":"2023-10-03T17:34:41.238033+0000","objects_scrubbed":0,"log_size":822,"log_dups_size":0,"ondisk_log_size":822,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:29:20.305630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":200,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1127,"num_read_kb":944,"num_write":42,"num_write_kb":42,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"46'31291","reported_seq":61917,"reported_epoch":46,"state":"active+clean","last_fresh":"2023-10-03T17:40:03.016378+0000","last_change":"2023-10-03T17:34:50.321236+0000","last_active":"2023-10-03T17:40:03.016378+0000","last_peered":"2023-10-03T17:40:03.016378+0000","last_clean":"2023-10-03T17:40:03.016378+0000","last_became_active":"2023-10-03T17:34:48.310555+0000","last_became_peered":"2023-10-03T17:34:48.310555+0000","last_unstale":"2023-10-03T17:40:03.016378+0000","last_undegraded":"2023-10-03T17:40:03.016378+0000","last_fullsized":"2023-10-03T17:40:03.016378+0000","mapping_epoch":16,"log_start":"44'21200","ondisk_log_start":"44'21200","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_clean_scrub_stamp":"2023-10-03T17:34:47.281095+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T19:53:40.901598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086289699999999995,"stat_sum":{"num_bytes":5000892416,"num_objects":1242,"num_object_clones":0,"num_object_copies":2484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1242,"num_whiteouts":0,"num_read":30561,"num_read_kb":107880,"num_write":31291,"num_write_kb":125164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"46'24","reported_seq":471,"reported_epoch":46,"state":"active+clean","last_fresh":"2023-10-03T17:40:01.861419+0000","last_change":"2023-10-03T17:34:42.278925+0000","last_active":"2023-10-03T17:40:01.861419+0000","last_peered":"2023-10-03T17:40:01.861419+0000","last_clean":"2023-10-03T17:40:01.861419+0000","last_became_active":"2023-10-03T17:34:42.278196+0000","last_became_peered":"2023-10-03T17:34:42.278196+0000","last_unstale":"2023-10-03T17:40:01.861419+0000","last_undegraded":"2023-10-03T17:40:01.861419+0000","last_fullsized":"2023-10-03T17:40:01.861419+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:34:41.238033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:34:41.238033+0000","last_clean_scrub_stamp":"2023-10-03T17:34:41.238033+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T00:36:18.073012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":823,"num_read_kb":694,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"46'32385","reported_seq":63458,"reported_epoch":46,"state":"active+clean","last_fresh":"2023-10-03T17:40:02.995867+0000","last_change":"2023-10-03T17:34:50.396806+0000","last_active":"2023-10-03T17:40:02.995867+0000","last_peered":"2023-10-03T17:40:02.995867+0000","last_clean":"2023-10-03T17:40:02.995867+0000","last_became_active":"2023-10-03T17:34:48.310912+0000","last_became_peered":"2023-10-03T17:34:48.310912+0000","last_unstale":"2023-10-03T17:40:02.995867+0000","last_undegraded":"2023-10-03T17:40:02.995867+0000","last_fullsized":"2023-10-03T17:40:02.995867+0000","mapping_epoch":16,"log_start":"44'22300","ondisk_log_start":"44'22300","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_clean_scrub_stamp":"2023-10-03T17:34:47.281095+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T18:44:12.843757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045661900000000001,"stat_sum":{"num_bytes":5124481024,"num_objects":1271,"num_object_clones":0,"num_object_copies":2542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1271,"num_whiteouts":0,"num_read":31006,"num_read_kb":110640,"num_write":32385,"num_write_kb":129540,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"46'13","reported_seq":211,"reported_epoch":46,"state":"active+clean","last_fresh":"2023-10-03T17:40:01.840287+0000","last_change":"2023-10-03T17:34:42.273043+0000","last_active":"2023-10-03T17:40:01.840287+0000","last_peered":"2023-10-03T17:40:01.840287+0000","last_clean":"2023-10-03T17:40:01.840287+0000","last_became_active":"2023-10-03T17:34:42.272306+0000","last_became_peered":"2023-10-03T17:34:42.272306+0000","last_unstale":"2023-10-03T17:40:01.840287+0000","last_undegraded":"2023-10-03T17:40:01.840287+0000","last_fullsized":"2023-10-03T17:40:01.840287+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:34:41.238033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:34:41.238033+0000","last_clean_scrub_stamp":"2023-10-03T17:34:41.238033+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T18:02:59.747348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":200,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":309,"num_read_kb":242,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"46'33258","reported_seq":65018,"reported_epoch":46,"state":"active+clean","last_fresh":"2023-10-03T17:40:03.017249+0000","last_change":"2023-10-03T17:34:50.322676+0000","last_active":"2023-10-03T17:40:03.017249+0000","last_peered":"2023-10-03T17:40:03.017249+0000","last_clean":"2023-10-03T17:40:03.017249+0000","last_became_active":"2023-10-03T17:34:48.310612+0000","last_became_peered":"2023-10-03T17:34:48.310612+0000","last_unstale":"2023-10-03T17:40:03.017249+0000","last_undegraded":"2023-10-03T17:40:03.017249+0000","last_fullsized":"2023-10-03T17:40:03.017249+0000","mapping_epoch":16,"log_start":"44'23200","ondisk_log_start":"44'23200","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_clean_scrub_stamp":"2023-10-03T17:34:47.281095+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T20:53:44.523856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022375120000000001,"stat_sum":{"num_bytes":5191618579,"num_objects":1288,"num_object_clones":0,"num_object_copies":2576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1288,"num_whiteouts":0,"num_read":31697,"num_read_kb":112600,"num_write":33258,"num_write_kb":133026,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"36'4","reported_seq":62,"reported_epoch":45,"state":"active+clean","last_fresh":"2023-10-03T17:38:55.232552+0000","last_change":"2023-10-03T17:34:42.271102+0000","last_active":"2023-10-03T17:38:55.232552+0000","last_peered":"2023-10-03T17:38:55.232552+0000","last_clean":"2023-10-03T17:38:55.232552+0000","last_became_active":"2023-10-03T17:34:42.270194+0000","last_became_peered":"2023-10-03T17:34:42.270194+0000","last_unstale":"2023-10-03T17:38:55.232552+0000","last_undegraded":"2023-10-03T17:38:55.232552+0000","last_fullsized":"2023-10-03T17:38:55.232552+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:34:41.238033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:34:41.238033+0000","last_clean_scrub_stamp":"2023-10-03T17:34:41.238033+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:41:26.441632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":132,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":8,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":174,"reported_epoch":45,"state":"active+clean","last_fresh":"2023-10-03T17:38:55.237074+0000","last_change":"2023-10-03T17:34:39.504748+0000","last_active":"2023-10-03T17:38:55.237074+0000","last_peered":"2023-10-03T17:38:55.237074+0000","last_clean":"2023-10-03T17:38:55.237074+0000","last_became_active":"2023-10-03T17:34:39.502393+0000","last_became_peered":"2023-10-03T17:34:39.502393+0000","last_unstale":"2023-10-03T17:38:55.237074+0000","last_undegraded":"2023-10-03T17:38:55.237074+0000","last_fullsized":"2023-10-03T17:38:55.237074+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:34:38.227481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:34:38.227481+0000","last_clean_scrub_stamp":"2023-10-03T17:34:38.227481+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T00:10:27.940656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":20634824723,"num_objects":5121,"num_object_clones":0,"num_object_copies":10242,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5121,"num_whiteouts":0,"num_read":125533,"num_read_kb":445648,"num_write":130626,"num_write_kb":522498,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1070002176,"data_stored":1069994022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40326,"ondisk_log_size":40326,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":1328,"num_objects":63,"num_object_clones":0,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63,"num_whiteouts":0,"num_read":6261,"num_read_kb":5277,"num_write":203,"num_write_kb":203,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":2656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3444,"ondisk_log_size":3444,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":7460,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738439,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":932344,"kb_used_data":522352,"kb_used_omap":1,"kb_used_meta":409982,"kb_avail":93439496,"statfs":{"total":96636764160,"available":95682043904,"internally_reserved":0,"allocated":534888448,"data_stored":534601244,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":419822022},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,3,2,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 17:39:44 2023","interfaces":[{"interface":"back","average":{"1min":0.375,"5min":0.45000000000000001,"15min":0.49399999999999999},"min":{"1min":0.182,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":1.982,"5min":4.4640000000000004,"15min":4.4640000000000004},"last":0.25800000000000001},{"interface":"front","average":{"1min":0.42499999999999999,"5min":0.41699999999999998,"15min":0.51000000000000001},"min":{"1min":0.19800000000000001,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":1.897,"5min":1.897,"15min":1.897},"last":0.27900000000000003}]},{"osd":1,"last update":"Tue Oct 3 17:39:44 2023","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.46700000000000003,"15min":0.52600000000000002},"min":{"1min":0.20300000000000001,"5min":0.16600000000000001,"15min":0.16600000000000001},"max":{"1min":2.2040000000000002,"5min":4.4260000000000002,"15min":4.4260000000000002},"last":0.22},{"interface":"front","average":{"1min":0.375,"5min":0.41699999999999998,"15min":0.54000000000000004},"min":{"1min":0.223,"5min":0.192,"15min":0.192},"max":{"1min":0.70499999999999996,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.23599999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738438,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":469716,"kb_used_data":258124,"kb_used_omap":1,"kb_used_meta":211582,"kb_avail":93902124,"statfs":{"total":96636764160,"available":96155774976,"internally_reserved":0,"allocated":264318976,"data_stored":264455083,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":216660422},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,0,1,4,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 17:39:56 2023","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.39200000000000002,"15min":0.47799999999999998},"min":{"1min":0.187,"5min":0.161,"15min":0.161},"max":{"1min":1.577,"5min":4.1879999999999997,"15min":4.1879999999999997},"last":0.218},{"interface":"front","average":{"1min":0.371,"5min":0.38700000000000001,"15min":0.48799999999999999},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":1.3899999999999999,"5min":1.532,"15min":1.532},"last":0.19900000000000001}]},{"osd":2,"last update":"Tue Oct 3 17:39:56 2023","interfaces":[{"interface":"back","average":{"1min":0.37,"5min":0.38700000000000001,"15min":0.49099999999999999},"min":{"1min":0.17799999999999999,"5min":0.159,"15min":0.159},"max":{"1min":2.4889999999999999,"5min":2.4889999999999999,"15min":2.4889999999999999},"last":3.0390000000000001},{"interface":"front","average":{"1min":0.39600000000000002,"5min":0.39700000000000002,"15min":0.51900000000000002},"min":{"1min":0.19400000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":2.5219999999999998,"5min":2.5219999999999998,"15min":2.5219999999999998},"last":3.0699999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738438,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":462264,"kb_used_data":264560,"kb_used_omap":1,"kb_used_meta":197694,"kb_avail":93909576,"statfs":{"total":96636764160,"available":96163405824,"internally_reserved":0,"allocated":270909440,"data_stored":271102423,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":202439110},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,2,3,10,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 17:39:43 2023","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.38400000000000001,"15min":0.42399999999999999},"min":{"1min":0.17299999999999999,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":3.2269999999999999,"5min":3.2269999999999999,"15min":3.2269999999999999},"last":0.19700000000000001},{"interface":"front","average":{"1min":0.28499999999999998,"5min":0.41499999999999998,"15min":0.45800000000000002},"min":{"1min":0.19,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.45800000000000002,"5min":2.1680000000000001,"15min":2.1680000000000001},"last":0.22600000000000001}]},{"osd":2,"last update":"Tue Oct 3 17:39:43 2023","interfaces":[{"interface":"back","average":{"1min":0.32300000000000001,"5min":0.46300000000000002,"15min":0.54300000000000004},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.61399999999999999,"5min":2.823,"15min":2.823},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.47099999999999997,"15min":0.52900000000000003},"min":{"1min":0.249,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":2.9239999999999999,"5min":2.9239999999999999,"15min":2.9239999999999999},"last":0.26100000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":879,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":270651392,"data_stored":270651392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":264388608,"data_stored":264384531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":534962176,"data_stored":534958099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-03T17:40:05.169 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-10-03T17:40:05.169 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T17:40:05.749 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-03T17:40:05.749 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-10-03T17:40:05.795 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":208,"stamp":"2023-10-03T17:40:04.025047+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20635416419,"num_objects":5186,"num_object_clones":0,"num_object_copies":10372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5186,"num_whiteouts":0,"num_read":131900,"num_read_kb":451016,"num_write":130966,"num_write_kb":524595,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43867,"ondisk_log_size":43867,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1864324,"kb_used_data":1045036,"kb_used_omap":4,"kb_used_meta":819259,"kb_avail":281251196,"statfs":{"total":289910292480,"available":288001224704,"internally_reserved":0,"allocated":1070116864,"data_stored":1070158750,"data_compressed":7460,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4782,"internal_metadata":838921554},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,3,0,6,9,15,9],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":65814528,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8885,"num_read_kb":31647,"num_write":7614,"num_write_kb":30456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006264"},"pg_stats":[{"pgid":"2.7","version":"46'632","reported_seq":1090,"reported_epoch":46,"state":"active+clean","last_fresh":"2023-10-03T17:40:01.867643+0000","last_change":"2023-10-03T17:34:42.277381+0000","last_active":"2023-10-03T17:40:01.867643+0000","last_peered":"2023-10-03T17:40:01.867643+0000","last_clean":"2023-10-03T17:40:01.867643+0000","last_became_active":"2023-10-03T17:34:42.276624+0000","last_became_peered":"2023-10-03T17:34:42.276624+0000","last_unstale":"2023-10-03T17:40:01.867643+0000","last_undegraded":"2023-10-03T17:40:01.867643+0000","last_fullsized":"2023-10-03T17:40:01.867643+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:34:41.238033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:34:41.238033+0000","last_clean_scrub_stamp":"2023-10-03T17:34:41.238033+0000","objects_scrubbed":0,"log_size":632,"log_dups_size":0,"ondisk_log_size":632,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T18:56:53.056366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":132,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":837,"num_read_kb":699,"num_write":16,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"46'1077","reported_seq":2044,"reported_epoch":46,"state":"active+clean","last_fresh":"2023-10-03T17:40:01.867049+0000","last_change":"2023-10-03T17:34:42.271628+0000","last_active":"2023-10-03T17:40:01.867049+0000","last_peered":"2023-10-03T17:40:01.867049+0000","last_clean":"2023-10-03T17:40:01.867049+0000","last_became_active":"2023-10-03T17:34:42.271201+0000","last_became_peered":"2023-10-03T17:34:42.271201+0000","last_unstale":"2023-10-03T17:40:01.867049+0000","last_undegraded":"2023-10-03T17:40:01.867049+0000","last_fullsized":"2023-10-03T17:40:01.867049+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:34:41.238033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:34:41.238033+0000","last_clean_scrub_stamp":"2023-10-03T17:34:41.238033+0000","objects_scrubbed":0,"log_size":1077,"log_dups_size":0,"ondisk_log_size":1077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:13:49.428992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":218,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1806,"num_read_kb":1593,"num_write":68,"num_write_kb":68,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"39'7","reported_seq":72,"reported_epoch":45,"state":"active+clean","last_fresh":"2023-10-03T17:38:55.232572+0000","last_change":"2023-10-03T17:34:42.277773+0000","last_active":"2023-10-03T17:38:55.232572+0000","last_peered":"2023-10-03T17:38:55.232572+0000","last_clean":"2023-10-03T17:38:55.232572+0000","last_became_active":"2023-10-03T17:34:42.276941+0000","last_became_peered":"2023-10-03T17:34:42.276941+0000","last_unstale":"2023-10-03T17:38:55.232572+0000","last_undegraded":"2023-10-03T17:38:55.232572+0000","last_fullsized":"2023-10-03T17:38:55.232572+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:34:41.238033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:34:41.238033+0000","last_clean_scrub_stamp":"2023-10-03T17:34:41.238033+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T21:07:05.154169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":249,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"46'865","reported_seq":1605,"reported_epoch":46,"state":"active+clean","last_fresh":"2023-10-03T17:40:01.869122+0000","last_change":"2023-10-03T17:34:42.276876+0000","last_active":"2023-10-03T17:40:01.869122+0000","last_peered":"2023-10-03T17:40:01.869122+0000","last_clean":"2023-10-03T17:40:01.869122+0000","last_became_active":"2023-10-03T17:34:42.276459+0000","last_became_peered":"2023-10-03T17:34:42.276459+0000","last_unstale":"2023-10-03T17:40:01.869122+0000","last_undegraded":"2023-10-03T17:40:01.869122+0000","last_fullsized":"2023-10-03T17:40:01.869122+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:34:41.238033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:34:41.238033+0000","last_clean_scrub_stamp":"2023-10-03T17:34:41.238033+0000","objects_scrubbed":0,"log_size":865,"log_dups_size":0,"ondisk_log_size":865,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:03:40.525005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":80,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":1294,"num_read_kb":1085,"num_write":41,"num_write_kb":41,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"46'33692","reported_seq":66023,"reported_epoch":46,"state":"active+clean","last_fresh":"2023-10-03T17:40:03.016087+0000","last_change":"2023-10-03T17:34:50.396952+0000","last_active":"2023-10-03T17:40:03.016087+0000","last_peered":"2023-10-03T17:40:03.016087+0000","last_clean":"2023-10-03T17:40:03.016087+0000","last_became_active":"2023-10-03T17:34:48.310701+0000","last_became_peered":"2023-10-03T17:34:48.310701+0000","last_unstale":"2023-10-03T17:40:03.016087+0000","last_undegraded":"2023-10-03T17:40:03.016087+0000","last_fullsized":"2023-10-03T17:40:03.016087+0000","mapping_epoch":16,"log_start":"44'23600","ondisk_log_start":"44'23600","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_clean_scrub_stamp":"2023-10-03T17:34:47.281095+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T21:26:47.697148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040768800000000002,"stat_sum":{"num_bytes":5317832704,"num_objects":1320,"num_object_clones":0,"num_object_copies":2640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1320,"num_whiteouts":0,"num_read":32269,"num_read_kb":114528,"num_write":33692,"num_write_kb":134768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"46'822","reported_seq":1427,"reported_epoch":46,"state":"active+clean","last_fresh":"2023-10-03T17:40:02.605566+0000","last_change":"2023-10-03T17:34:42.271479+0000","last_active":"2023-10-03T17:40:02.605566+0000","last_peered":"2023-10-03T17:40:02.605566+0000","last_clean":"2023-10-03T17:40:02.605566+0000","last_became_active":"2023-10-03T17:34:42.270625+0000","last_became_peered":"2023-10-03T17:34:42.270625+0000","last_unstale":"2023-10-03T17:40:02.605566+0000","last_undegraded":"2023-10-03T17:40:02.605566+0000","last_fullsized":"2023-10-03T17:40:02.605566+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:34:41.238033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:34:41.238033+0000","last_clean_scrub_stamp":"2023-10-03T17:34:41.238033+0000","objects_scrubbed":0,"log_size":822,"log_dups_size":0,"ondisk_log_size":822,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:29:20.305630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":200,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1127,"num_read_kb":944,"num_write":42,"num_write_kb":42,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"46'31291","reported_seq":61917,"reported_epoch":46,"state":"active+clean","last_fresh":"2023-10-03T17:40:03.016378+0000","last_change":"2023-10-03T17:34:50.321236+0000","last_active":"2023-10-03T17:40:03.016378+0000","last_peered":"2023-10-03T17:40:03.016378+0000","last_clean":"2023-10-03T17:40:03.016378+0000","last_became_active":"2023-10-03T17:34:48.310555+0000","last_became_peered":"2023-10-03T17:34:48.310555+0000","last_unstale":"2023-10-03T17:40:03.016378+0000","last_undegraded":"2023-10-03T17:40:03.016378+0000","last_fullsized":"2023-10-03T17:40:03.016378+0000","mapping_epoch":16,"log_start":"44'21200","ondisk_log_start":"44'21200","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_clean_scrub_stamp":"2023-10-03T17:34:47.281095+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T19:53:40.901598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086289699999999995,"stat_sum":{"num_bytes":5000892416,"num_objects":1242,"num_object_clones":0,"num_object_copies":2484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1242,"num_whiteouts":0,"num_read":30561,"num_read_kb":107880,"num_write":31291,"num_write_kb":125164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"46'24","reported_seq":471,"reported_epoch":46,"state":"active+clean","last_fresh":"2023-10-03T17:40:01.861419+0000","last_change":"2023-10-03T17:34:42.278925+0000","last_active":"2023-10-03T17:40:01.861419+0000","last_peered":"2023-10-03T17:40:01.861419+0000","last_clean":"2023-10-03T17:40:01.861419+0000","last_became_active":"2023-10-03T17:34:42.278196+0000","last_became_peered":"2023-10-03T17:34:42.278196+0000","last_unstale":"2023-10-03T17:40:01.861419+0000","last_undegraded":"2023-10-03T17:40:01.861419+0000","last_fullsized":"2023-10-03T17:40:01.861419+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:34:41.238033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:34:41.238033+0000","last_clean_scrub_stamp":"2023-10-03T17:34:41.238033+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T00:36:18.073012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":823,"num_read_kb":694,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"46'32385","reported_seq":63458,"reported_epoch":46,"state":"active+clean","last_fresh":"2023-10-03T17:40:02.995867+0000","last_change":"2023-10-03T17:34:50.396806+0000","last_active":"2023-10-03T17:40:02.995867+0000","last_peered":"2023-10-03T17:40:02.995867+0000","last_clean":"2023-10-03T17:40:02.995867+0000","last_became_active":"2023-10-03T17:34:48.310912+0000","last_became_peered":"2023-10-03T17:34:48.310912+0000","last_unstale":"2023-10-03T17:40:02.995867+0000","last_undegraded":"2023-10-03T17:40:02.995867+0000","last_fullsized":"2023-10-03T17:40:02.995867+0000","mapping_epoch":16,"log_start":"44'22300","ondisk_log_start":"44'22300","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_clean_scrub_stamp":"2023-10-03T17:34:47.281095+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T18:44:12.843757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045661900000000001,"stat_sum":{"num_bytes":5124481024,"num_objects":1271,"num_object_clones":0,"num_object_copies":2542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1271,"num_whiteouts":0,"num_read":31006,"num_read_kb":110640,"num_write":32385,"num_write_kb":129540,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"46'13","reported_seq":211,"reported_epoch":46,"state":"active+clean","last_fresh":"2023-10-03T17:40:01.840287+0000","last_change":"2023-10-03T17:34:42.273043+0000","last_active":"2023-10-03T17:40:01.840287+0000","last_peered":"2023-10-03T17:40:01.840287+0000","last_clean":"2023-10-03T17:40:01.840287+0000","last_became_active":"2023-10-03T17:34:42.272306+0000","last_became_peered":"2023-10-03T17:34:42.272306+0000","last_unstale":"2023-10-03T17:40:01.840287+0000","last_undegraded":"2023-10-03T17:40:01.840287+0000","last_fullsized":"2023-10-03T17:40:01.840287+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:34:41.238033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:34:41.238033+0000","last_clean_scrub_stamp":"2023-10-03T17:34:41.238033+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T18:02:59.747348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":200,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":309,"num_read_kb":242,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"46'33258","reported_seq":65018,"reported_epoch":46,"state":"active+clean","last_fresh":"2023-10-03T17:40:03.017249+0000","last_change":"2023-10-03T17:34:50.322676+0000","last_active":"2023-10-03T17:40:03.017249+0000","last_peered":"2023-10-03T17:40:03.017249+0000","last_clean":"2023-10-03T17:40:03.017249+0000","last_became_active":"2023-10-03T17:34:48.310612+0000","last_became_peered":"2023-10-03T17:34:48.310612+0000","last_unstale":"2023-10-03T17:40:03.017249+0000","last_undegraded":"2023-10-03T17:40:03.017249+0000","last_fullsized":"2023-10-03T17:40:03.017249+0000","mapping_epoch":16,"log_start":"44'23200","ondisk_log_start":"44'23200","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_clean_scrub_stamp":"2023-10-03T17:34:47.281095+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T20:53:44.523856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022375120000000001,"stat_sum":{"num_bytes":5191618579,"num_objects":1288,"num_object_clones":0,"num_object_copies":2576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1288,"num_whiteouts":0,"num_read":31697,"num_read_kb":112600,"num_write":33258,"num_write_kb":133026,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"36'4","reported_seq":62,"reported_epoch":45,"state":"active+clean","last_fresh":"2023-10-03T17:38:55.232552+0000","last_change":"2023-10-03T17:34:42.271102+0000","last_active":"2023-10-03T17:38:55.232552+0000","last_peered":"2023-10-03T17:38:55.232552+0000","last_clean":"2023-10-03T17:38:55.232552+0000","last_became_active":"2023-10-03T17:34:42.270194+0000","last_became_peered":"2023-10-03T17:34:42.270194+0000","last_unstale":"2023-10-03T17:38:55.232552+0000","last_undegraded":"2023-10-03T17:38:55.232552+0000","last_fullsized":"2023-10-03T17:38:55.232552+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:34:41.238033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:34:41.238033+0000","last_clean_scrub_stamp":"2023-10-03T17:34:41.238033+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:41:26.441632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":132,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":8,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":174,"reported_epoch":45,"state":"active+clean","last_fresh":"2023-10-03T17:38:55.237074+0000","last_change":"2023-10-03T17:34:39.504748+0000","last_active":"2023-10-03T17:38:55.237074+0000","last_peered":"2023-10-03T17:38:55.237074+0000","last_clean":"2023-10-03T17:38:55.237074+0000","last_became_active":"2023-10-03T17:34:39.502393+0000","last_became_peered":"2023-10-03T17:34:39.502393+0000","last_unstale":"2023-10-03T17:38:55.237074+0000","last_undegraded":"2023-10-03T17:38:55.237074+0000","last_fullsized":"2023-10-03T17:38:55.237074+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:34:38.227481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:34:38.227481+0000","last_clean_scrub_stamp":"2023-10-03T17:34:38.227481+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T00:10:27.940656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":20634824723,"num_objects":5121,"num_object_clones":0,"num_object_copies":10242,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5121,"num_whiteouts":0,"num_read":125533,"num_read_kb":445648,"num_write":130626,"num_write_kb":522498,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1070002176,"data_stored":1069994022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40326,"ondisk_log_size":40326,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":1328,"num_objects":63,"num_object_clones":0,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63,"num_whiteouts":0,"num_read":6261,"num_read_kb":5277,"num_write":203,"num_write_kb":203,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":2656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3444,"ondisk_log_size":3444,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":7460,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738439,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":932344,"kb_used_data":522352,"kb_used_omap":1,"kb_used_meta":409982,"kb_avail":93439496,"statfs":{"total":96636764160,"available":95682043904,"internally_reserved":0,"allocated":534888448,"data_stored":534601244,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":419822022},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,3,2,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 17:39:44 2023","interfaces":[{"interface":"back","average":{"1min":0.375,"5min":0.45000000000000001,"15min":0.49399999999999999},"min":{"1min":0.182,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":1.982,"5min":4.4640000000000004,"15min":4.4640000000000004},"last":0.25800000000000001},{"interface":"front","average":{"1min":0.42499999999999999,"5min":0.41699999999999998,"15min":0.51000000000000001},"min":{"1min":0.19800000000000001,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":1.897,"5min":1.897,"15min":1.897},"last":0.27900000000000003}]},{"osd":1,"last update":"Tue Oct 3 17:39:44 2023","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.46700000000000003,"15min":0.52600000000000002},"min":{"1min":0.20300000000000001,"5min":0.16600000000000001,"15min":0.16600000000000001},"max":{"1min":2.2040000000000002,"5min":4.4260000000000002,"15min":4.4260000000000002},"last":0.22},{"interface":"front","average":{"1min":0.375,"5min":0.41699999999999998,"15min":0.54000000000000004},"min":{"1min":0.223,"5min":0.192,"15min":0.192},"max":{"1min":0.70499999999999996,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.23599999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738438,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":469716,"kb_used_data":258124,"kb_used_omap":1,"kb_used_meta":211582,"kb_avail":93902124,"statfs":{"total":96636764160,"available":96155774976,"internally_reserved":0,"allocated":264318976,"data_stored":264455083,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":216660422},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,0,1,4,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 17:39:56 2023","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.39200000000000002,"15min":0.47799999999999998},"min":{"1min":0.187,"5min":0.161,"15min":0.161},"max":{"1min":1.577,"5min":4.1879999999999997,"15min":4.1879999999999997},"last":0.218},{"interface":"front","average":{"1min":0.371,"5min":0.38700000000000001,"15min":0.48799999999999999},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":1.3899999999999999,"5min":1.532,"15min":1.532},"last":0.19900000000000001}]},{"osd":2,"last update":"Tue Oct 3 17:39:56 2023","interfaces":[{"interface":"back","average":{"1min":0.37,"5min":0.38700000000000001,"15min":0.49099999999999999},"min":{"1min":0.17799999999999999,"5min":0.159,"15min":0.159},"max":{"1min":2.4889999999999999,"5min":2.4889999999999999,"15min":2.4889999999999999},"last":3.0390000000000001},{"interface":"front","average":{"1min":0.39600000000000002,"5min":0.39700000000000002,"15min":0.51900000000000002},"min":{"1min":0.19400000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":2.5219999999999998,"5min":2.5219999999999998,"15min":2.5219999999999998},"last":3.0699999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738438,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":462264,"kb_used_data":264560,"kb_used_omap":1,"kb_used_meta":197694,"kb_avail":93909576,"statfs":{"total":96636764160,"available":96163405824,"internally_reserved":0,"allocated":270909440,"data_stored":271102423,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":202439110},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,2,3,10,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 17:39:43 2023","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.38400000000000001,"15min":0.42399999999999999},"min":{"1min":0.17299999999999999,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":3.2269999999999999,"5min":3.2269999999999999,"15min":3.2269999999999999},"last":0.19700000000000001},{"interface":"front","average":{"1min":0.28499999999999998,"5min":0.41499999999999998,"15min":0.45800000000000002},"min":{"1min":0.19,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.45800000000000002,"5min":2.1680000000000001,"15min":2.1680000000000001},"last":0.22600000000000001}]},{"osd":2,"last update":"Tue Oct 3 17:39:43 2023","interfaces":[{"interface":"back","average":{"1min":0.32300000000000001,"5min":0.46300000000000002,"15min":0.54300000000000004},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.61399999999999999,"5min":2.823,"15min":2.823},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.47099999999999997,"15min":0.52900000000000003},"min":{"1min":0.249,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":2.9239999999999999,"5min":2.9239999999999999,"15min":2.9239999999999999},"last":0.26100000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":879,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":270651392,"data_stored":270651392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":264388608,"data_stored":264384531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":534962176,"data_stored":534958099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-03T17:40:05.797 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-03T17:40:06.328 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-03T17:40:06.328 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":46,"fsid":"499d6abd-e417-4d8d-a4aa-a7db52cf5eb1","created":"2023-10-03T17:34:25.157173+0000","modified":"2023-10-03T17:38:56.219586+0000","last_up_change":"2023-10-03T17:34:36.222314+0000","last_in_change":"2023-10-03T17:34:26.068746+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-03T17:34:38.029646+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-03T17:34:40.410622+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"datapool","create_time":"2023-10-03T17:34:46.318619+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"40","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":23,"snap_epoch":40,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7f9ffbc3-1e8c-44e2-8305-abc2b594dbd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":1646836731},{"type":"v1","addr":"172.21.15.111:6809","nonce":1646836731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":1646836731},{"type":"v1","addr":"172.21.15.111:6811","nonce":1646836731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":1646836731},{"type":"v1","addr":"172.21.15.111:6815","nonce":1646836731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":1646836731},{"type":"v1","addr":"172.21.15.111:6813","nonce":1646836731}]},"public_addr":"172.21.15.111:6809/1646836731","cluster_addr":"172.21.15.111:6811/1646836731","heartbeat_back_addr":"172.21.15.111:6815/1646836731","heartbeat_front_addr":"172.21.15.111:6813/1646836731","state":["exists","up"]},{"osd":1,"uuid":"92793af8-c508-44bb-918d-7a8a92862c82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":3440296436},{"type":"v1","addr":"172.21.15.111:6817","nonce":3440296436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":3440296436},{"type":"v1","addr":"172.21.15.111:6819","nonce":3440296436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":3440296436},{"type":"v1","addr":"172.21.15.111:6823","nonce":3440296436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":3440296436},{"type":"v1","addr":"172.21.15.111:6821","nonce":3440296436}]},"public_addr":"172.21.15.111:6817/3440296436","cluster_addr":"172.21.15.111:6819/3440296436","heartbeat_back_addr":"172.21.15.111:6823/3440296436","heartbeat_front_addr":"172.21.15.111:6821/3440296436","state":["exists","up"]},{"osd":2,"uuid":"354eb2cb-387e-4416-a5b0-c80ffd0bd292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":653857722},{"type":"v1","addr":"172.21.15.111:6801","nonce":653857722}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":653857722},{"type":"v1","addr":"172.21.15.111:6803","nonce":653857722}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":653857722},{"type":"v1","addr":"172.21.15.111:6807","nonce":653857722}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":653857722},{"type":"v1","addr":"172.21.15.111:6805","nonce":653857722}]},"public_addr":"172.21.15.111:6801/653857722","cluster_addr":"172.21.15.111:6803/653857722","heartbeat_back_addr":"172.21.15.111:6807/653857722","heartbeat_front_addr":"172.21.15.111:6805/653857722","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-03T17:34:31.339197+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-03T17:34:31.634807+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-03T17:34:31.432310+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.111:0/4228986752":"2023-10-03T18:38:54.497203+0000","172.21.15.111:0/262203629":"2023-10-03T18:37:46.084918+0000","172.21.15.111:0/3950478362":"2023-10-03T18:36:37.300973+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-03T17:40:07.371 INFO:tasks.ceph:Scrubbing osd.0 2023-10-03T17:40:07.371 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2023-10-03T17:40:07.634 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2023-10-03T17:40:07.634 INFO:teuthology.orchestra.run.smithi111.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-03T17:40:07.634 INFO:teuthology.orchestra.run.smithi111.stdout:} 2023-10-03T17:40:07.664 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2023-10-03T17:40:08.129 INFO:teuthology.orchestra.run.smithi111.stderr:instructed osd(s) 0 to deep-scrub 2023-10-03T17:40:08.161 INFO:tasks.ceph:Scrubbing osd.1 2023-10-03T17:40:08.162 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2023-10-03T17:40:08.419 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2023-10-03T17:40:08.420 INFO:teuthology.orchestra.run.smithi111.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-03T17:40:08.420 INFO:teuthology.orchestra.run.smithi111.stdout:} 2023-10-03T17:40:08.463 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2023-10-03T17:40:09.094 INFO:teuthology.orchestra.run.smithi111.stderr:instructed osd(s) 1 to deep-scrub 2023-10-03T17:40:09.126 INFO:tasks.ceph:Scrubbing osd.2 2023-10-03T17:40:09.126 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2023-10-03T17:40:09.401 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2023-10-03T17:40:09.401 INFO:teuthology.orchestra.run.smithi111.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-03T17:40:09.401 INFO:teuthology.orchestra.run.smithi111.stdout:} 2023-10-03T17:40:09.431 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2023-10-03T17:40:10.012 INFO:teuthology.orchestra.run.smithi111.stderr:instructed osd(s) 2 to deep-scrub 2023-10-03T17:40:10.052 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T17:40:10.614 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-03T17:40:10.615 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-10-03T17:40:10.657 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":211,"stamp":"2023-10-03T17:40:10.030266+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20655744867,"num_objects":5186,"num_object_clones":0,"num_object_copies":10372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5186,"num_whiteouts":0,"num_read":134278,"num_read_kb":460184,"num_write":133356,"num_write_kb":534155,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43837,"ondisk_log_size":43837,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1900100,"kb_used_data":1064172,"kb_used_omap":4,"kb_used_meta":835899,"kb_avail":281215420,"statfs":{"total":289910292480,"available":287964590080,"internally_reserved":0,"allocated":1089712128,"data_stored":1089754014,"data_compressed":7460,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4782,"internal_metadata":855960914},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,3,6,12,17,16],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":43905024,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5791,"num_read_kb":19887,"num_write":4894,"num_write_kb":19576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.010584"},"pg_stats":[{"pgid":"2.7","version":"46'650","reported_seq":1111,"reported_epoch":46,"state":"active+clean","last_fresh":"2023-10-03T17:40:07.591642+0000","last_change":"2023-10-03T17:34:42.277381+0000","last_active":"2023-10-03T17:40:07.591642+0000","last_peered":"2023-10-03T17:40:07.591642+0000","last_clean":"2023-10-03T17:40:07.591642+0000","last_became_active":"2023-10-03T17:34:42.276624+0000","last_became_peered":"2023-10-03T17:34:42.276624+0000","last_unstale":"2023-10-03T17:40:07.591642+0000","last_undegraded":"2023-10-03T17:40:07.591642+0000","last_fullsized":"2023-10-03T17:40:07.591642+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:34:41.238033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:34:41.238033+0000","last_clean_scrub_stamp":"2023-10-03T17:34:41.238033+0000","objects_scrubbed":0,"log_size":650,"log_dups_size":0,"ondisk_log_size":650,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T18:56:53.056366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":132,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":837,"num_read_kb":699,"num_write":16,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"46'1107","reported_seq":2079,"reported_epoch":46,"state":"active+clean","last_fresh":"2023-10-03T17:40:07.595130+0000","last_change":"2023-10-03T17:34:42.271628+0000","last_active":"2023-10-03T17:40:07.595130+0000","last_peered":"2023-10-03T17:40:07.595130+0000","last_clean":"2023-10-03T17:40:07.595130+0000","last_became_active":"2023-10-03T17:34:42.271201+0000","last_became_peered":"2023-10-03T17:34:42.271201+0000","last_unstale":"2023-10-03T17:40:07.595130+0000","last_undegraded":"2023-10-03T17:40:07.595130+0000","last_fullsized":"2023-10-03T17:40:07.595130+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:34:41.238033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:34:41.238033+0000","last_clean_scrub_stamp":"2023-10-03T17:34:41.238033+0000","objects_scrubbed":0,"log_size":1107,"log_dups_size":0,"ondisk_log_size":1107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:13:49.428992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":218,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1806,"num_read_kb":1593,"num_write":68,"num_write_kb":68,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"39'7","reported_seq":72,"reported_epoch":45,"state":"active+clean","last_fresh":"2023-10-03T17:38:55.232572+0000","last_change":"2023-10-03T17:34:42.277773+0000","last_active":"2023-10-03T17:38:55.232572+0000","last_peered":"2023-10-03T17:38:55.232572+0000","last_clean":"2023-10-03T17:38:55.232572+0000","last_became_active":"2023-10-03T17:34:42.276941+0000","last_became_peered":"2023-10-03T17:34:42.276941+0000","last_unstale":"2023-10-03T17:38:55.232572+0000","last_undegraded":"2023-10-03T17:38:55.232572+0000","last_fullsized":"2023-10-03T17:38:55.232572+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:34:41.238033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:34:41.238033+0000","last_clean_scrub_stamp":"2023-10-03T17:34:41.238033+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T21:07:05.154169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":249,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"46'889","reported_seq":1633,"reported_epoch":46,"state":"active+clean","last_fresh":"2023-10-03T17:40:07.596015+0000","last_change":"2023-10-03T17:34:42.276876+0000","last_active":"2023-10-03T17:40:07.596015+0000","last_peered":"2023-10-03T17:40:07.596015+0000","last_clean":"2023-10-03T17:40:07.596015+0000","last_became_active":"2023-10-03T17:34:42.276459+0000","last_became_peered":"2023-10-03T17:34:42.276459+0000","last_unstale":"2023-10-03T17:40:07.596015+0000","last_undegraded":"2023-10-03T17:40:07.596015+0000","last_fullsized":"2023-10-03T17:40:07.596015+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:34:41.238033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:34:41.238033+0000","last_clean_scrub_stamp":"2023-10-03T17:34:41.238033+0000","objects_scrubbed":0,"log_size":889,"log_dups_size":0,"ondisk_log_size":889,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:03:40.525005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":80,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":1294,"num_read_kb":1085,"num_write":41,"num_write_kb":41,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"46'34315","reported_seq":67236,"reported_epoch":46,"state":"active+clean","last_fresh":"2023-10-03T17:40:08.029003+0000","last_change":"2023-10-03T17:34:50.396952+0000","last_active":"2023-10-03T17:40:08.029003+0000","last_peered":"2023-10-03T17:40:08.029003+0000","last_clean":"2023-10-03T17:40:08.029003+0000","last_became_active":"2023-10-03T17:34:48.310701+0000","last_became_peered":"2023-10-03T17:34:48.310701+0000","last_unstale":"2023-10-03T17:40:08.029003+0000","last_undegraded":"2023-10-03T17:40:08.029003+0000","last_fullsized":"2023-10-03T17:40:08.029003+0000","mapping_epoch":16,"log_start":"44'24300","ondisk_log_start":"44'24300","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_clean_scrub_stamp":"2023-10-03T17:34:47.281095+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T21:26:47.697148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040768800000000002,"stat_sum":{"num_bytes":5321445376,"num_objects":1320,"num_object_clones":0,"num_object_copies":2640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1320,"num_whiteouts":0,"num_read":32865,"num_read_kb":116824,"num_write":34315,"num_write_kb":137260,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"46'830","reported_seq":1439,"reported_epoch":46,"state":"active+clean","last_fresh":"2023-10-03T17:40:07.735786+0000","last_change":"2023-10-03T17:34:42.271479+0000","last_active":"2023-10-03T17:40:07.735786+0000","last_peered":"2023-10-03T17:40:07.735786+0000","last_clean":"2023-10-03T17:40:07.735786+0000","last_became_active":"2023-10-03T17:34:42.270625+0000","last_became_peered":"2023-10-03T17:34:42.270625+0000","last_unstale":"2023-10-03T17:40:07.735786+0000","last_undegraded":"2023-10-03T17:40:07.735786+0000","last_fullsized":"2023-10-03T17:40:07.735786+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:34:41.238033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:34:41.238033+0000","last_clean_scrub_stamp":"2023-10-03T17:34:41.238033+0000","objects_scrubbed":0,"log_size":830,"log_dups_size":0,"ondisk_log_size":830,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:29:20.305630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":200,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1127,"num_read_kb":944,"num_write":42,"num_write_kb":42,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"46'31872","reported_seq":63073,"reported_epoch":46,"state":"active+clean","last_fresh":"2023-10-03T17:40:08.031672+0000","last_change":"2023-10-03T17:34:50.321236+0000","last_active":"2023-10-03T17:40:08.031672+0000","last_peered":"2023-10-03T17:40:08.031672+0000","last_clean":"2023-10-03T17:40:08.031672+0000","last_became_active":"2023-10-03T17:34:48.310555+0000","last_became_peered":"2023-10-03T17:34:48.310555+0000","last_unstale":"2023-10-03T17:40:08.031672+0000","last_undegraded":"2023-10-03T17:40:08.031672+0000","last_fullsized":"2023-10-03T17:40:08.031672+0000","mapping_epoch":16,"log_start":"44'21800","ondisk_log_start":"44'21800","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_clean_scrub_stamp":"2023-10-03T17:34:47.281095+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T19:53:40.901598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086289699999999995,"stat_sum":{"num_bytes":5008601088,"num_objects":1242,"num_object_clones":0,"num_object_copies":2484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1242,"num_whiteouts":0,"num_read":31137,"num_read_kb":110080,"num_write":31872,"num_write_kb":127488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"46'24","reported_seq":474,"reported_epoch":46,"state":"active+clean","last_fresh":"2023-10-03T17:40:03.773358+0000","last_change":"2023-10-03T17:34:42.278925+0000","last_active":"2023-10-03T17:40:03.773358+0000","last_peered":"2023-10-03T17:40:03.773358+0000","last_clean":"2023-10-03T17:40:03.773358+0000","last_became_active":"2023-10-03T17:34:42.278196+0000","last_became_peered":"2023-10-03T17:34:42.278196+0000","last_unstale":"2023-10-03T17:40:03.773358+0000","last_undegraded":"2023-10-03T17:40:03.773358+0000","last_fullsized":"2023-10-03T17:40:03.773358+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:34:41.238033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:34:41.238033+0000","last_clean_scrub_stamp":"2023-10-03T17:34:41.238033+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T00:36:18.073012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":823,"num_read_kb":694,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"46'33000","reported_seq":64662,"reported_epoch":46,"state":"active+clean","last_fresh":"2023-10-03T17:40:08.023366+0000","last_change":"2023-10-03T17:34:50.396806+0000","last_active":"2023-10-03T17:40:08.023366+0000","last_peered":"2023-10-03T17:40:08.023366+0000","last_clean":"2023-10-03T17:40:08.023366+0000","last_became_active":"2023-10-03T17:34:48.310912+0000","last_became_peered":"2023-10-03T17:34:48.310912+0000","last_unstale":"2023-10-03T17:40:08.023366+0000","last_undegraded":"2023-10-03T17:40:08.023366+0000","last_fullsized":"2023-10-03T17:40:08.023366+0000","mapping_epoch":16,"log_start":"44'22900","ondisk_log_start":"44'22900","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_clean_scrub_stamp":"2023-10-03T17:34:47.281095+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T18:44:12.843757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045661900000000001,"stat_sum":{"num_bytes":5129060352,"num_objects":1271,"num_object_clones":0,"num_object_copies":2542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1271,"num_whiteouts":0,"num_read":31597,"num_read_kb":112912,"num_write":33000,"num_write_kb":132000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"46'13","reported_seq":212,"reported_epoch":46,"state":"active+clean","last_fresh":"2023-10-03T17:40:01.840880+0000","last_change":"2023-10-03T17:34:42.273043+0000","last_active":"2023-10-03T17:40:01.840880+0000","last_peered":"2023-10-03T17:40:01.840880+0000","last_clean":"2023-10-03T17:40:01.840880+0000","last_became_active":"2023-10-03T17:34:42.272306+0000","last_became_peered":"2023-10-03T17:34:42.272306+0000","last_unstale":"2023-10-03T17:40:01.840880+0000","last_undegraded":"2023-10-03T17:40:01.840880+0000","last_fullsized":"2023-10-03T17:40:01.840880+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:34:41.238033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:34:41.238033+0000","last_clean_scrub_stamp":"2023-10-03T17:34:41.238033+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T18:02:59.747348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":200,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":309,"num_read_kb":242,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"46'33829","reported_seq":66208,"reported_epoch":46,"state":"active+clean","last_fresh":"2023-10-03T17:40:08.023099+0000","last_change":"2023-10-03T17:34:50.322676+0000","last_active":"2023-10-03T17:40:08.023099+0000","last_peered":"2023-10-03T17:40:08.023099+0000","last_clean":"2023-10-03T17:40:08.023099+0000","last_became_active":"2023-10-03T17:34:48.310612+0000","last_became_peered":"2023-10-03T17:34:48.310612+0000","last_unstale":"2023-10-03T17:40:08.023099+0000","last_undegraded":"2023-10-03T17:40:08.023099+0000","last_fullsized":"2023-10-03T17:40:08.023099+0000","mapping_epoch":16,"log_start":"44'23800","ondisk_log_start":"44'23800","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_clean_scrub_stamp":"2023-10-03T17:34:47.281095+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T20:53:44.523856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022375120000000001,"stat_sum":{"num_bytes":5196046355,"num_objects":1288,"num_object_clones":0,"num_object_copies":2576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1288,"num_whiteouts":0,"num_read":32312,"num_read_kb":115000,"num_write":33829,"num_write_kb":135310,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"36'4","reported_seq":62,"reported_epoch":45,"state":"active+clean","last_fresh":"2023-10-03T17:38:55.232552+0000","last_change":"2023-10-03T17:34:42.271102+0000","last_active":"2023-10-03T17:38:55.232552+0000","last_peered":"2023-10-03T17:38:55.232552+0000","last_clean":"2023-10-03T17:38:55.232552+0000","last_became_active":"2023-10-03T17:34:42.270194+0000","last_became_peered":"2023-10-03T17:34:42.270194+0000","last_unstale":"2023-10-03T17:38:55.232552+0000","last_undegraded":"2023-10-03T17:38:55.232552+0000","last_fullsized":"2023-10-03T17:38:55.232552+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:34:41.238033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:34:41.238033+0000","last_clean_scrub_stamp":"2023-10-03T17:34:41.238033+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:41:26.441632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":132,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":8,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":174,"reported_epoch":45,"state":"active+clean","last_fresh":"2023-10-03T17:38:55.237074+0000","last_change":"2023-10-03T17:34:39.504748+0000","last_active":"2023-10-03T17:38:55.237074+0000","last_peered":"2023-10-03T17:38:55.237074+0000","last_clean":"2023-10-03T17:38:55.237074+0000","last_became_active":"2023-10-03T17:34:39.502393+0000","last_became_peered":"2023-10-03T17:34:39.502393+0000","last_unstale":"2023-10-03T17:38:55.237074+0000","last_undegraded":"2023-10-03T17:38:55.237074+0000","last_fullsized":"2023-10-03T17:38:55.237074+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:34:38.227481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:34:38.227481+0000","last_clean_scrub_stamp":"2023-10-03T17:34:38.227481+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T00:10:27.940656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":20655153171,"num_objects":5121,"num_object_clones":0,"num_object_copies":10242,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5121,"num_whiteouts":0,"num_read":127911,"num_read_kb":454816,"num_write":133016,"num_write_kb":532058,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1089630208,"data_stored":1089622054,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40216,"ondisk_log_size":40216,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":1328,"num_objects":63,"num_object_clones":0,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63,"num_whiteouts":0,"num_read":6261,"num_read_kb":5277,"num_write":203,"num_write_kb":203,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":2656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3524,"ondisk_log_size":3524,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":7460,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738440,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":950164,"kb_used_data":531852,"kb_used_omap":1,"kb_used_meta":418302,"kb_avail":93421676,"statfs":{"total":96636764160,"available":95663796224,"internally_reserved":0,"allocated":544616448,"data_stored":544329244,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":428341702},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,1,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 17:39:44 2023","interfaces":[{"interface":"back","average":{"1min":0.375,"5min":0.45000000000000001,"15min":0.49399999999999999},"min":{"1min":0.182,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":1.982,"5min":4.4640000000000004,"15min":4.4640000000000004},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.42499999999999999,"5min":0.41699999999999998,"15min":0.51000000000000001},"min":{"1min":0.19800000000000001,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":1.897,"5min":1.897,"15min":1.897},"last":0.40000000000000002}]},{"osd":1,"last update":"Tue Oct 3 17:39:44 2023","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.46700000000000003,"15min":0.52600000000000002},"min":{"1min":0.20300000000000001,"5min":0.16600000000000001,"15min":0.16600000000000001},"max":{"1min":2.2040000000000002,"5min":4.4260000000000002,"15min":4.4260000000000002},"last":0.246},{"interface":"front","average":{"1min":0.375,"5min":0.41699999999999998,"15min":0.54000000000000004},"min":{"1min":0.223,"5min":0.192,"15min":0.192},"max":{"1min":0.70499999999999996,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.433}]}]},{"osd":1,"up_from":8,"seq":34359738439,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":478668,"kb_used_data":262916,"kb_used_omap":1,"kb_used_meta":215742,"kb_avail":93893172,"statfs":{"total":96636764160,"available":96146608128,"internally_reserved":0,"allocated":269225984,"data_stored":269362091,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":220920262},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,0,5,12,10],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 17:39:56 2023","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.39200000000000002,"15min":0.47799999999999998},"min":{"1min":0.187,"5min":0.161,"15min":0.161},"max":{"1min":1.577,"5min":4.1879999999999997,"15min":4.1879999999999997},"last":0.218},{"interface":"front","average":{"1min":0.371,"5min":0.38700000000000001,"15min":0.48799999999999999},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":1.3899999999999999,"5min":1.532,"15min":1.532},"last":0.19900000000000001}]},{"osd":2,"last update":"Tue Oct 3 17:39:56 2023","interfaces":[{"interface":"back","average":{"1min":0.37,"5min":0.38700000000000001,"15min":0.49099999999999999},"min":{"1min":0.17799999999999999,"5min":0.159,"15min":0.159},"max":{"1min":2.4889999999999999,"5min":2.4889999999999999,"15min":2.4889999999999999},"last":0.28299999999999997},{"interface":"front","average":{"1min":0.39600000000000002,"5min":0.39700000000000002,"15min":0.51900000000000002},"min":{"1min":0.19400000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":2.5219999999999998,"5min":2.5219999999999998,"15min":2.5219999999999998},"last":0.26500000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738439,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":471268,"kb_used_data":269404,"kb_used_omap":1,"kb_used_meta":201854,"kb_avail":93900572,"statfs":{"total":96636764160,"available":96154185728,"internally_reserved":0,"allocated":275869696,"data_stored":276062679,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":206698950},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,3,6,2,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 17:39:43 2023","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.38400000000000001,"15min":0.42399999999999999},"min":{"1min":0.17299999999999999,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":3.2269999999999999,"5min":3.2269999999999999,"15min":3.2269999999999999},"last":0.215},{"interface":"front","average":{"1min":0.28499999999999998,"5min":0.41499999999999998,"15min":0.45800000000000002},"min":{"1min":0.19,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.45800000000000002,"5min":2.1680000000000001,"15min":2.1680000000000001},"last":0.22900000000000001}]},{"osd":2,"last update":"Tue Oct 3 17:39:43 2023","interfaces":[{"interface":"back","average":{"1min":0.32300000000000001,"5min":0.46300000000000002,"15min":0.54300000000000004},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.61399999999999999,"5min":2.823,"15min":2.823},"last":0.307},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.47099999999999997,"15min":0.52900000000000003},"min":{"1min":0.249,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":2.9239999999999999,"5min":2.9239999999999999,"15min":2.9239999999999999},"last":0.38800000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":879,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":275734528,"data_stored":275734528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":269127680,"data_stored":269123603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":544768000,"data_stored":544763923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-03T17:40:10.661 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-03T17:34:41.238033+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=34, tm_sec=41, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=40, tm_sec=6, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T17:40:10.661 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-03T17:34:41.238033+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=34, tm_sec=41, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=40, tm_sec=6, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T17:40:10.661 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-03T17:34:41.238033+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=34, tm_sec=41, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=40, tm_sec=6, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T17:40:10.661 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-03T17:34:41.238033+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=34, tm_sec=41, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=40, tm_sec=6, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T17:40:10.662 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-03T17:34:47.281095+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=34, tm_sec=47, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=40, tm_sec=6, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T17:40:10.662 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-10-03T17:34:41.238033+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=34, tm_sec=41, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=40, tm_sec=6, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T17:40:10.662 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-03T17:34:47.281095+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=34, tm_sec=47, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=40, tm_sec=6, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T17:40:10.663 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-10-03T17:34:41.238033+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=34, tm_sec=41, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=40, tm_sec=6, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T17:40:10.663 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-03T17:34:47.281095+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=34, tm_sec=47, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=40, tm_sec=6, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T17:40:10.663 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-10-03T17:34:41.238033+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=34, tm_sec=41, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=40, tm_sec=6, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T17:40:10.663 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-03T17:34:47.281095+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=34, tm_sec=47, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=40, tm_sec=6, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T17:40:10.664 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-03T17:34:41.238033+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=34, tm_sec=41, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=40, tm_sec=6, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T17:40:10.664 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2023-10-03T17:34:38.227481+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=34, tm_sec=38, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=40, tm_sec=6, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T17:40:10.664 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T17:40:30.665 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T17:40:31.296 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-03T17:40:31.298 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-10-03T17:40:31.341 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":221,"stamp":"2023-10-03T17:40:30.033422+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20707247971,"num_objects":5186,"num_object_clones":0,"num_object_copies":10372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5186,"num_whiteouts":0,"num_read":143892,"num_read_kb":497396,"num_write":142995,"num_write_kb":572711,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44231,"ondisk_log_size":44231,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2004072,"kb_used_data":1140368,"kb_used_omap":4,"kb_used_meta":863675,"kb_avail":281111448,"statfs":{"total":289910292480,"available":287858122752,"internally_reserved":0,"allocated":1167736832,"data_stored":1167778718,"data_compressed":7460,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4782,"internal_metadata":884403538},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,3,4,3,7,23],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":32305152,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7278,"num_read_kb":28140,"num_write":7253,"num_write_kb":29012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6449,"num_omap_keys":233,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001824"},"pg_stats":[{"pgid":"2.7","version":"46'712","reported_seq":1194,"reported_epoch":46,"state":"active+clean","last_fresh":"2023-10-03T17:40:27.798057+0000","last_change":"2023-10-03T17:40:17.407011+0000","last_active":"2023-10-03T17:40:27.798057+0000","last_peered":"2023-10-03T17:40:27.798057+0000","last_clean":"2023-10-03T17:40:27.798057+0000","last_became_active":"2023-10-03T17:34:42.276624+0000","last_became_peered":"2023-10-03T17:34:42.276624+0000","last_unstale":"2023-10-03T17:40:27.798057+0000","last_undegraded":"2023-10-03T17:40:27.798057+0000","last_fullsized":"2023-10-03T17:40:27.798057+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'673","last_scrub_stamp":"2023-10-03T17:40:17.406813+0000","last_deep_scrub":"46'673","last_deep_scrub_stamp":"2023-10-03T17:40:17.406813+0000","last_clean_scrub_stamp":"2023-10-03T17:40:17.406813+0000","objects_scrubbed":7,"log_size":712,"log_dups_size":0,"ondisk_log_size":712,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T21:38:35.056889+0000","scrub_duration":0.013778076,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":132,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":837,"num_read_kb":699,"num_write":16,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":903,"num_omap_keys":30,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"46'1209","reported_seq":2210,"reported_epoch":46,"state":"active+clean","last_fresh":"2023-10-03T17:40:27.797972+0000","last_change":"2023-10-03T17:40:13.376519+0000","last_active":"2023-10-03T17:40:27.797972+0000","last_peered":"2023-10-03T17:40:27.797972+0000","last_clean":"2023-10-03T17:40:27.797972+0000","last_became_active":"2023-10-03T17:34:42.271201+0000","last_became_peered":"2023-10-03T17:34:42.271201+0000","last_unstale":"2023-10-03T17:40:27.797972+0000","last_undegraded":"2023-10-03T17:40:27.797972+0000","last_fullsized":"2023-10-03T17:40:27.797972+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'1129","last_scrub_stamp":"2023-10-03T17:40:13.376332+0000","last_deep_scrub":"46'1129","last_deep_scrub_stamp":"2023-10-03T17:40:13.376332+0000","last_clean_scrub_stamp":"2023-10-03T17:40:13.376332+0000","objects_scrubbed":11,"log_size":1209,"log_dups_size":0,"ondisk_log_size":1209,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T18:59:17.088585+0000","scrub_duration":0.015998675,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":218,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1806,"num_read_kb":1593,"num_write":68,"num_write_kb":68,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2912,"num_omap_keys":93,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"46'14","reported_seq":88,"reported_epoch":46,"state":"active+clean","last_fresh":"2023-10-03T17:40:25.495935+0000","last_change":"2023-10-03T17:40:25.496010+0000","last_active":"2023-10-03T17:40:25.495935+0000","last_peered":"2023-10-03T17:40:25.495935+0000","last_clean":"2023-10-03T17:40:25.495935+0000","last_became_active":"2023-10-03T17:34:42.276941+0000","last_became_peered":"2023-10-03T17:34:42.276941+0000","last_unstale":"2023-10-03T17:40:25.495935+0000","last_undegraded":"2023-10-03T17:40:25.495935+0000","last_fullsized":"2023-10-03T17:40:25.495935+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'14","last_scrub_stamp":"2023-10-03T17:40:25.495827+0000","last_deep_scrub":"46'14","last_deep_scrub_stamp":"2023-10-03T17:40:25.495827+0000","last_clean_scrub_stamp":"2023-10-03T17:40:25.495827+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:05:44.939663+0000","scrub_duration":0.0133078,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":249,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"46'972","reported_seq":1741,"reported_epoch":46,"state":"active+clean","last_fresh":"2023-10-03T17:40:27.796057+0000","last_change":"2023-10-03T17:40:26.513902+0000","last_active":"2023-10-03T17:40:27.796057+0000","last_peered":"2023-10-03T17:40:27.796057+0000","last_clean":"2023-10-03T17:40:27.796057+0000","last_became_active":"2023-10-03T17:34:42.276459+0000","last_became_peered":"2023-10-03T17:34:42.276459+0000","last_unstale":"2023-10-03T17:40:27.796057+0000","last_undegraded":"2023-10-03T17:40:27.796057+0000","last_fullsized":"2023-10-03T17:40:27.796057+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'964","last_scrub_stamp":"2023-10-03T17:40:26.513729+0000","last_deep_scrub":"46'964","last_deep_scrub_stamp":"2023-10-03T17:40:26.513729+0000","last_clean_scrub_stamp":"2023-10-03T17:40:26.513729+0000","objects_scrubbed":10,"log_size":972,"log_dups_size":0,"ondisk_log_size":972,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:41:33.182751+0000","scrub_duration":0.018190280999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":80,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":1294,"num_read_kb":1085,"num_write":41,"num_write_kb":41,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1731,"num_omap_keys":80,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"46'36780","reported_seq":72227,"reported_epoch":46,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T17:40:28.122659+0000","last_change":"2023-10-03T17:40:11.716583+0000","last_active":"2023-10-03T17:40:28.122659+0000","last_peered":"2023-10-03T17:40:28.122659+0000","last_clean":"2023-10-03T17:40:28.122659+0000","last_became_active":"2023-10-03T17:34:48.310701+0000","last_became_peered":"2023-10-03T17:34:48.310701+0000","last_unstale":"2023-10-03T17:40:28.122659+0000","last_undegraded":"2023-10-03T17:40:28.122659+0000","last_fullsized":"2023-10-03T17:40:28.122659+0000","mapping_epoch":16,"log_start":"46'26700","ondisk_log_start":"46'26700","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_clean_scrub_stamp":"2023-10-03T17:34:47.281095+0000","objects_scrubbed":122,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 16s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040768800000000002,"stat_sum":{"num_bytes":5330968576,"num_objects":1320,"num_object_clones":0,"num_object_copies":2640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1320,"num_whiteouts":0,"num_read":35360,"num_read_kb":126460,"num_write":36775,"num_write_kb":147100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"46'901","reported_seq":1535,"reported_epoch":46,"state":"active+clean","last_fresh":"2023-10-03T17:40:26.123380+0000","last_change":"2023-10-03T17:40:08.675119+0000","last_active":"2023-10-03T17:40:26.123380+0000","last_peered":"2023-10-03T17:40:26.123380+0000","last_clean":"2023-10-03T17:40:26.123380+0000","last_became_active":"2023-10-03T17:34:42.270625+0000","last_became_peered":"2023-10-03T17:34:42.270625+0000","last_unstale":"2023-10-03T17:40:26.123380+0000","last_undegraded":"2023-10-03T17:40:26.123380+0000","last_fullsized":"2023-10-03T17:40:26.123380+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'841","last_scrub_stamp":"2023-10-03T17:40:08.674910+0000","last_deep_scrub":"46'841","last_deep_scrub_stamp":"2023-10-03T17:40:08.674910+0000","last_clean_scrub_stamp":"2023-10-03T17:40:08.674910+0000","objects_scrubbed":11,"log_size":901,"log_dups_size":0,"ondisk_log_size":901,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T21:05:42.790614+0000","scrub_duration":0.039189555000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":200,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1127,"num_read_kb":944,"num_write":42,"num_write_kb":42,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2564,"num_omap_keys":60,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"46'34244","reported_seq":67841,"reported_epoch":46,"state":"active+clean","last_fresh":"2023-10-03T17:40:28.092567+0000","last_change":"2023-10-03T17:34:50.321236+0000","last_active":"2023-10-03T17:40:28.092567+0000","last_peered":"2023-10-03T17:40:28.092567+0000","last_clean":"2023-10-03T17:40:28.092567+0000","last_became_active":"2023-10-03T17:34:48.310555+0000","last_became_peered":"2023-10-03T17:34:48.310555+0000","last_unstale":"2023-10-03T17:40:28.092567+0000","last_undegraded":"2023-10-03T17:40:28.092567+0000","last_fullsized":"2023-10-03T17:40:28.092567+0000","mapping_epoch":16,"log_start":"46'24200","ondisk_log_start":"46'24200","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_clean_scrub_stamp":"2023-10-03T17:34:47.281095+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086289699999999995,"stat_sum":{"num_bytes":5023412224,"num_objects":1242,"num_object_clones":0,"num_object_copies":2484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1242,"num_whiteouts":0,"num_read":33530,"num_read_kb":119336,"num_write":34244,"num_write_kb":136976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"46'30","reported_seq":501,"reported_epoch":46,"state":"active+clean","last_fresh":"2023-10-03T17:40:24.268378+0000","last_change":"2023-10-03T17:40:17.641493+0000","last_active":"2023-10-03T17:40:24.268378+0000","last_peered":"2023-10-03T17:40:24.268378+0000","last_clean":"2023-10-03T17:40:24.268378+0000","last_became_active":"2023-10-03T17:34:42.278196+0000","last_became_peered":"2023-10-03T17:34:42.278196+0000","last_unstale":"2023-10-03T17:40:24.268378+0000","last_undegraded":"2023-10-03T17:40:24.268378+0000","last_fullsized":"2023-10-03T17:40:24.268378+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'30","last_scrub_stamp":"2023-10-03T17:40:17.640969+0000","last_deep_scrub":"46'30","last_deep_scrub_stamp":"2023-10-03T17:40:17.640969+0000","last_clean_scrub_stamp":"2023-10-03T17:40:17.640969+0000","objects_scrubbed":6,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:04:02.954896+0000","scrub_duration":0.071054565,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":823,"num_read_kb":694,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":903,"num_omap_keys":30,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"46'35381","reported_seq":69553,"reported_epoch":46,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T17:40:28.122402+0000","last_change":"2023-10-03T17:40:10.712199+0000","last_active":"2023-10-03T17:40:28.122402+0000","last_peered":"2023-10-03T17:40:28.122402+0000","last_clean":"2023-10-03T17:40:28.122402+0000","last_became_active":"2023-10-03T17:34:48.310912+0000","last_became_peered":"2023-10-03T17:34:48.310912+0000","last_unstale":"2023-10-03T17:40:28.122402+0000","last_undegraded":"2023-10-03T17:40:28.122402+0000","last_fullsized":"2023-10-03T17:40:28.122402+0000","mapping_epoch":16,"log_start":"46'25300","ondisk_log_start":"46'25300","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_clean_scrub_stamp":"2023-10-03T17:34:47.281095+0000","objects_scrubbed":101,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 17s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045661900000000001,"stat_sum":{"num_bytes":5142642688,"num_objects":1271,"num_object_clones":0,"num_object_copies":2542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1271,"num_whiteouts":0,"num_read":34084,"num_read_kb":122528,"num_write":35375,"num_write_kb":141500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"46'20","reported_seq":232,"reported_epoch":46,"state":"active+clean","last_fresh":"2023-10-03T17:40:19.217699+0000","last_change":"2023-10-03T17:40:10.708344+0000","last_active":"2023-10-03T17:40:19.217699+0000","last_peered":"2023-10-03T17:40:19.217699+0000","last_clean":"2023-10-03T17:40:19.217699+0000","last_became_active":"2023-10-03T17:34:42.272306+0000","last_became_peered":"2023-10-03T17:34:42.272306+0000","last_unstale":"2023-10-03T17:40:19.217699+0000","last_undegraded":"2023-10-03T17:40:19.217699+0000","last_fullsized":"2023-10-03T17:40:19.217699+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'20","last_scrub_stamp":"2023-10-03T17:40:10.708060+0000","last_deep_scrub":"46'20","last_deep_scrub_stamp":"2023-10-03T17:40:10.708060+0000","last_clean_scrub_stamp":"2023-10-03T17:40:10.708060+0000","objects_scrubbed":7,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:03:09.097252+0000","scrub_duration":0.042279491000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":200,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":309,"num_read_kb":242,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":301,"num_omap_keys":10,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"46'36261","reported_seq":70879,"reported_epoch":46,"state":"active+clean","last_fresh":"2023-10-03T17:40:28.101417+0000","last_change":"2023-10-03T17:34:50.322676+0000","last_active":"2023-10-03T17:40:28.101417+0000","last_peered":"2023-10-03T17:40:28.101417+0000","last_clean":"2023-10-03T17:40:28.101417+0000","last_became_active":"2023-10-03T17:34:48.310612+0000","last_became_peered":"2023-10-03T17:34:48.310612+0000","last_unstale":"2023-10-03T17:40:28.101417+0000","last_undegraded":"2023-10-03T17:40:28.101417+0000","last_fullsized":"2023-10-03T17:40:28.101417+0000","mapping_epoch":16,"log_start":"46'26200","ondisk_log_start":"46'26200","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_clean_scrub_stamp":"2023-10-03T17:34:47.281095+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022375120000000001,"stat_sum":{"num_bytes":5209632787,"num_objects":1288,"num_object_clones":0,"num_object_copies":2576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1288,"num_whiteouts":0,"num_read":34551,"num_read_kb":123704,"num_write":36261,"num_write_kb":145038,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"46'8","reported_seq":75,"reported_epoch":46,"state":"active+clean","last_fresh":"2023-10-03T17:40:24.472668+0000","last_change":"2023-10-03T17:40:24.472734+0000","last_active":"2023-10-03T17:40:24.472668+0000","last_peered":"2023-10-03T17:40:24.472668+0000","last_clean":"2023-10-03T17:40:24.472668+0000","last_became_active":"2023-10-03T17:34:42.270194+0000","last_became_peered":"2023-10-03T17:34:42.270194+0000","last_unstale":"2023-10-03T17:40:24.472668+0000","last_undegraded":"2023-10-03T17:40:24.472668+0000","last_fullsized":"2023-10-03T17:40:24.472668+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'8","last_scrub_stamp":"2023-10-03T17:40:24.472573+0000","last_deep_scrub":"46'8","last_deep_scrub_stamp":"2023-10-03T17:40:24.472573+0000","last_clean_scrub_stamp":"2023-10-03T17:40:24.472573+0000","objects_scrubbed":4,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T18:28:04.735033+0000","scrub_duration":0.024990637999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":132,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":8,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"46'99","reported_seq":185,"reported_epoch":46,"state":"active+clean","last_fresh":"2023-10-03T17:40:10.449177+0000","last_change":"2023-10-03T17:40:10.452341+0000","last_active":"2023-10-03T17:40:10.449177+0000","last_peered":"2023-10-03T17:40:10.449177+0000","last_clean":"2023-10-03T17:40:10.449177+0000","last_became_active":"2023-10-03T17:34:39.502393+0000","last_became_peered":"2023-10-03T17:34:39.502393+0000","last_unstale":"2023-10-03T17:40:10.449177+0000","last_undegraded":"2023-10-03T17:40:10.449177+0000","last_fullsized":"2023-10-03T17:40:10.449177+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'99","last_scrub_stamp":"2023-10-03T17:40:10.449099+0000","last_deep_scrub":"46'99","last_deep_scrub_stamp":"2023-10-03T17:40:10.449099+0000","last_clean_scrub_stamp":"2023-10-03T17:40:10.449099+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T20:18:04.958666+0000","scrub_duration":0.043225012,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":20706656275,"num_objects":5121,"num_object_clones":0,"num_object_copies":10242,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5121,"num_whiteouts":0,"num_read":137525,"num_read_kb":492028,"num_write":142655,"num_write_kb":570614,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1168601088,"data_stored":1168592934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40266,"ondisk_log_size":40266,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":1328,"num_objects":63,"num_object_clones":0,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63,"num_whiteouts":0,"num_read":6261,"num_read_kb":5277,"num_write":203,"num_write_kb":203,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":2656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3866,"ondisk_log_size":3866,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":7460,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738444,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1024024,"kb_used_data":569616,"kb_used_omap":1,"kb_used_meta":454398,"kb_avail":93347816,"statfs":{"total":96636764160,"available":95588163584,"internally_reserved":0,"allocated":583286784,"data_stored":582999580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":465304006},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 17:39:44 2023","interfaces":[{"interface":"back","average":{"1min":0.375,"5min":0.45000000000000001,"15min":0.49399999999999999},"min":{"1min":0.182,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":1.982,"5min":4.4640000000000004,"15min":4.4640000000000004},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.42499999999999999,"5min":0.41699999999999998,"15min":0.51000000000000001},"min":{"1min":0.19800000000000001,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":1.897,"5min":1.897,"15min":1.897},"last":0.30599999999999999}]},{"osd":1,"last update":"Tue Oct 3 17:39:44 2023","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.46700000000000003,"15min":0.52600000000000002},"min":{"1min":0.20300000000000001,"5min":0.16600000000000001,"15min":0.16600000000000001},"max":{"1min":2.2040000000000002,"5min":4.4260000000000002,"15min":4.4260000000000002},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.375,"5min":0.41699999999999998,"15min":0.54000000000000004},"min":{"1min":0.223,"5min":0.192,"15min":0.192},"max":{"1min":0.70499999999999996,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.34699999999999998}]}]},{"osd":1,"up_from":8,"seq":34359738443,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":501916,"kb_used_data":282004,"kb_used_omap":1,"kb_used_meta":219902,"kb_avail":93869924,"statfs":{"total":96636764160,"available":96122802176,"internally_reserved":0,"allocated":288772096,"data_stored":288908203,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":225180102},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,0,1,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 17:39:56 2023","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.39200000000000002,"15min":0.47799999999999998},"min":{"1min":0.187,"5min":0.161,"15min":0.161},"max":{"1min":1.577,"5min":4.1879999999999997,"15min":4.1879999999999997},"last":0.24399999999999999},{"interface":"front","average":{"1min":0.371,"5min":0.38700000000000001,"15min":0.48799999999999999},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":1.3899999999999999,"5min":1.532,"15min":1.532},"last":0.29799999999999999}]},{"osd":2,"last update":"Tue Oct 3 17:39:56 2023","interfaces":[{"interface":"back","average":{"1min":0.37,"5min":0.38700000000000001,"15min":0.49099999999999999},"min":{"1min":0.17799999999999999,"5min":0.159,"15min":0.159},"max":{"1min":2.4889999999999999,"5min":2.4889999999999999,"15min":2.4889999999999999},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.39600000000000002,"5min":0.39700000000000002,"15min":0.51900000000000002},"min":{"1min":0.19400000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":2.5219999999999998,"5min":2.5219999999999998,"15min":2.5219999999999998},"last":0.28399999999999997}]}]},{"osd":0,"up_from":8,"seq":34359738443,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":478132,"kb_used_data":288748,"kb_used_omap":1,"kb_used_meta":189374,"kb_avail":93893708,"statfs":{"total":96636764160,"available":96147156992,"internally_reserved":0,"allocated":295677952,"data_stored":295870935,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":193919430},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,1,1,2,4,19],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 17:39:43 2023","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.38400000000000001,"15min":0.42399999999999999},"min":{"1min":0.17299999999999999,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":3.2269999999999999,"5min":3.2269999999999999,"15min":3.2269999999999999},"last":5.593},{"interface":"front","average":{"1min":0.28499999999999998,"5min":0.41499999999999998,"15min":0.45800000000000002},"min":{"1min":0.19,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.45800000000000002,"5min":2.1680000000000001,"15min":2.1680000000000001},"last":0.38}]},{"osd":2,"last update":"Tue Oct 3 17:39:43 2023","interfaces":[{"interface":"back","average":{"1min":0.32300000000000001,"5min":0.46300000000000002,"15min":0.54300000000000004},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.61399999999999999,"5min":2.823,"15min":2.823},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.47099999999999997,"15min":0.52900000000000003},"min":{"1min":0.249,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":2.9239999999999999,"5min":2.9239999999999999,"15min":2.9239999999999999},"last":0.29599999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":879,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":295530496,"data_stored":295530496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":288788480,"data_stored":288784403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":584282112,"data_stored":584278035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-03T17:40:31.343 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-03T17:34:47.281095+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=34, tm_sec=47, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=40, tm_sec=6, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T17:40:31.343 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-03T17:34:47.281095+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=34, tm_sec=47, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=40, tm_sec=6, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T17:40:31.343 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-03T17:34:47.281095+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=34, tm_sec=47, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=40, tm_sec=6, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T17:40:31.344 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-03T17:34:47.281095+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=34, tm_sec=47, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=40, tm_sec=6, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T17:40:31.344 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T17:40:51.345 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T17:40:51.900 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-03T17:40:51.901 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-10-03T17:40:51.930 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":231,"stamp":"2023-10-03T17:40:50.036621+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20729808739,"num_objects":5186,"num_object_clones":0,"num_object_copies":10372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5186,"num_whiteouts":0,"num_read":153045,"num_read_kb":532288,"num_write":152297,"num_write_kb":609919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44347,"ondisk_log_size":44347,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2123788,"kb_used_data":1214388,"kb_used_omap":4,"kb_used_meta":909371,"kb_avail":280991732,"statfs":{"total":289910292480,"available":287735533568,"internally_reserved":0,"allocated":1243533312,"data_stored":1243575198,"data_compressed":7460,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4782,"internal_metadata":931196242},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,2,1,5,51,7,8,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":16035840,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6853,"num_read_kb":26140,"num_write":6980,"num_write_kb":27920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001829"},"pg_stats":[{"pgid":"2.7","version":"46'766","reported_seq":1260,"reported_epoch":46,"state":"active+clean","last_fresh":"2023-10-03T17:40:46.110342+0000","last_change":"2023-10-03T17:40:17.407011+0000","last_active":"2023-10-03T17:40:46.110342+0000","last_peered":"2023-10-03T17:40:46.110342+0000","last_clean":"2023-10-03T17:40:46.110342+0000","last_became_active":"2023-10-03T17:34:42.276624+0000","last_became_peered":"2023-10-03T17:34:42.276624+0000","last_unstale":"2023-10-03T17:40:46.110342+0000","last_undegraded":"2023-10-03T17:40:46.110342+0000","last_fullsized":"2023-10-03T17:40:46.110342+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'673","last_scrub_stamp":"2023-10-03T17:40:17.406813+0000","last_deep_scrub":"46'673","last_deep_scrub_stamp":"2023-10-03T17:40:17.406813+0000","last_clean_scrub_stamp":"2023-10-03T17:40:17.406813+0000","objects_scrubbed":7,"log_size":766,"log_dups_size":0,"ondisk_log_size":766,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T21:38:35.056889+0000","scrub_duration":0.013778076,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":132,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":837,"num_read_kb":699,"num_write":16,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":903,"num_omap_keys":30,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"46'1299","reported_seq":2320,"reported_epoch":46,"state":"active+clean","last_fresh":"2023-10-03T17:40:46.113511+0000","last_change":"2023-10-03T17:40:13.376519+0000","last_active":"2023-10-03T17:40:46.113511+0000","last_peered":"2023-10-03T17:40:46.113511+0000","last_clean":"2023-10-03T17:40:46.113511+0000","last_became_active":"2023-10-03T17:34:42.271201+0000","last_became_peered":"2023-10-03T17:34:42.271201+0000","last_unstale":"2023-10-03T17:40:46.113511+0000","last_undegraded":"2023-10-03T17:40:46.113511+0000","last_fullsized":"2023-10-03T17:40:46.113511+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'1129","last_scrub_stamp":"2023-10-03T17:40:13.376332+0000","last_deep_scrub":"46'1129","last_deep_scrub_stamp":"2023-10-03T17:40:13.376332+0000","last_clean_scrub_stamp":"2023-10-03T17:40:13.376332+0000","objects_scrubbed":11,"log_size":1299,"log_dups_size":0,"ondisk_log_size":1299,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T18:59:17.088585+0000","scrub_duration":0.015998675,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":218,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1806,"num_read_kb":1593,"num_write":68,"num_write_kb":68,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2912,"num_omap_keys":93,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"46'14","reported_seq":88,"reported_epoch":46,"state":"active+clean","last_fresh":"2023-10-03T17:40:25.495935+0000","last_change":"2023-10-03T17:40:25.496010+0000","last_active":"2023-10-03T17:40:25.495935+0000","last_peered":"2023-10-03T17:40:25.495935+0000","last_clean":"2023-10-03T17:40:25.495935+0000","last_became_active":"2023-10-03T17:34:42.276941+0000","last_became_peered":"2023-10-03T17:34:42.276941+0000","last_unstale":"2023-10-03T17:40:25.495935+0000","last_undegraded":"2023-10-03T17:40:25.495935+0000","last_fullsized":"2023-10-03T17:40:25.495935+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'14","last_scrub_stamp":"2023-10-03T17:40:25.495827+0000","last_deep_scrub":"46'14","last_deep_scrub_stamp":"2023-10-03T17:40:25.495827+0000","last_clean_scrub_stamp":"2023-10-03T17:40:25.495827+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:05:44.939663+0000","scrub_duration":0.0133078,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":249,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"46'1044","reported_seq":1829,"reported_epoch":46,"state":"active+clean","last_fresh":"2023-10-03T17:40:46.110909+0000","last_change":"2023-10-03T17:40:26.513902+0000","last_active":"2023-10-03T17:40:46.110909+0000","last_peered":"2023-10-03T17:40:46.110909+0000","last_clean":"2023-10-03T17:40:46.110909+0000","last_became_active":"2023-10-03T17:34:42.276459+0000","last_became_peered":"2023-10-03T17:34:42.276459+0000","last_unstale":"2023-10-03T17:40:46.110909+0000","last_undegraded":"2023-10-03T17:40:46.110909+0000","last_fullsized":"2023-10-03T17:40:46.110909+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'964","last_scrub_stamp":"2023-10-03T17:40:26.513729+0000","last_deep_scrub":"46'964","last_deep_scrub_stamp":"2023-10-03T17:40:26.513729+0000","last_clean_scrub_stamp":"2023-10-03T17:40:26.513729+0000","objects_scrubbed":10,"log_size":1044,"log_dups_size":0,"ondisk_log_size":1044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:41:33.182751+0000","scrub_duration":0.018190280999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":80,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":1294,"num_read_kb":1085,"num_write":41,"num_write_kb":41,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1731,"num_omap_keys":80,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"46'39131","reported_seq":76982,"reported_epoch":46,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T17:40:48.245542+0000","last_change":"2023-10-03T17:40:11.716583+0000","last_active":"2023-10-03T17:40:48.245542+0000","last_peered":"2023-10-03T17:40:48.245542+0000","last_clean":"2023-10-03T17:40:48.245542+0000","last_became_active":"2023-10-03T17:34:48.310701+0000","last_became_peered":"2023-10-03T17:34:48.310701+0000","last_unstale":"2023-10-03T17:40:48.245542+0000","last_undegraded":"2023-10-03T17:40:48.245542+0000","last_fullsized":"2023-10-03T17:40:48.245542+0000","mapping_epoch":16,"log_start":"46'29100","ondisk_log_start":"46'29100","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_clean_scrub_stamp":"2023-10-03T17:34:47.281095+0000","objects_scrubbed":306,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 36s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040768800000000002,"stat_sum":{"num_bytes":5337448448,"num_objects":1320,"num_object_clones":0,"num_object_copies":2640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1320,"num_whiteouts":0,"num_read":37744,"num_read_kb":135520,"num_write":39112,"num_write_kb":156448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"46'969","reported_seq":1619,"reported_epoch":46,"state":"active+clean","last_fresh":"2023-10-03T17:40:47.678657+0000","last_change":"2023-10-03T17:40:08.675119+0000","last_active":"2023-10-03T17:40:47.678657+0000","last_peered":"2023-10-03T17:40:47.678657+0000","last_clean":"2023-10-03T17:40:47.678657+0000","last_became_active":"2023-10-03T17:34:42.270625+0000","last_became_peered":"2023-10-03T17:34:42.270625+0000","last_unstale":"2023-10-03T17:40:47.678657+0000","last_undegraded":"2023-10-03T17:40:47.678657+0000","last_fullsized":"2023-10-03T17:40:47.678657+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'841","last_scrub_stamp":"2023-10-03T17:40:08.674910+0000","last_deep_scrub":"46'841","last_deep_scrub_stamp":"2023-10-03T17:40:08.674910+0000","last_clean_scrub_stamp":"2023-10-03T17:40:08.674910+0000","objects_scrubbed":11,"log_size":969,"log_dups_size":0,"ondisk_log_size":969,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T21:05:42.790614+0000","scrub_duration":0.039189555000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":200,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1127,"num_read_kb":944,"num_write":42,"num_write_kb":42,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2564,"num_omap_keys":60,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"46'36524","reported_seq":72325,"reported_epoch":46,"state":"active+clean","last_fresh":"2023-10-03T17:40:48.176179+0000","last_change":"2023-10-03T17:40:43.628522+0000","last_active":"2023-10-03T17:40:48.176179+0000","last_peered":"2023-10-03T17:40:48.176179+0000","last_clean":"2023-10-03T17:40:48.176179+0000","last_became_active":"2023-10-03T17:34:48.310555+0000","last_became_peered":"2023-10-03T17:34:48.310555+0000","last_unstale":"2023-10-03T17:40:48.176179+0000","last_undegraded":"2023-10-03T17:40:48.176179+0000","last_fullsized":"2023-10-03T17:40:48.176179+0000","mapping_epoch":16,"log_start":"46'26500","ondisk_log_start":"46'26500","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_clean_scrub_stamp":"2023-10-03T17:34:47.281095+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086289699999999995,"stat_sum":{"num_bytes":5029400576,"num_objects":1242,"num_object_clones":0,"num_object_copies":2484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1242,"num_whiteouts":0,"num_read":35734,"num_read_kb":127768,"num_write":36524,"num_write_kb":146096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"46'30","reported_seq":513,"reported_epoch":46,"state":"active+clean","last_fresh":"2023-10-03T17:40:44.726318+0000","last_change":"2023-10-03T17:40:17.641493+0000","last_active":"2023-10-03T17:40:44.726318+0000","last_peered":"2023-10-03T17:40:44.726318+0000","last_clean":"2023-10-03T17:40:44.726318+0000","last_became_active":"2023-10-03T17:34:42.278196+0000","last_became_peered":"2023-10-03T17:34:42.278196+0000","last_unstale":"2023-10-03T17:40:44.726318+0000","last_undegraded":"2023-10-03T17:40:44.726318+0000","last_fullsized":"2023-10-03T17:40:44.726318+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'30","last_scrub_stamp":"2023-10-03T17:40:17.640969+0000","last_deep_scrub":"46'30","last_deep_scrub_stamp":"2023-10-03T17:40:17.640969+0000","last_clean_scrub_stamp":"2023-10-03T17:40:17.640969+0000","objects_scrubbed":6,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:04:02.954896+0000","scrub_duration":0.071054565,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":823,"num_read_kb":694,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":903,"num_omap_keys":30,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"46'37720","reported_seq":74218,"reported_epoch":46,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T17:40:48.232201+0000","last_change":"2023-10-03T17:40:10.712199+0000","last_active":"2023-10-03T17:40:48.232201+0000","last_peered":"2023-10-03T17:40:48.232201+0000","last_clean":"2023-10-03T17:40:48.232201+0000","last_became_active":"2023-10-03T17:34:48.310912+0000","last_became_peered":"2023-10-03T17:34:48.310912+0000","last_unstale":"2023-10-03T17:40:48.232201+0000","last_undegraded":"2023-10-03T17:40:48.232201+0000","last_fullsized":"2023-10-03T17:40:48.232201+0000","mapping_epoch":16,"log_start":"46'27700","ondisk_log_start":"46'27700","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_clean_scrub_stamp":"2023-10-03T17:34:47.281095+0000","objects_scrubbed":297,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 37s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045661900000000001,"stat_sum":{"num_bytes":5147566080,"num_objects":1271,"num_object_clones":0,"num_object_copies":2542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1271,"num_whiteouts":0,"num_read":36387,"num_read_kb":131320,"num_write":37707,"num_write_kb":150828,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"46'20","reported_seq":237,"reported_epoch":46,"state":"active+clean","last_fresh":"2023-10-03T17:40:43.145263+0000","last_change":"2023-10-03T17:40:10.708344+0000","last_active":"2023-10-03T17:40:43.145263+0000","last_peered":"2023-10-03T17:40:43.145263+0000","last_clean":"2023-10-03T17:40:43.145263+0000","last_became_active":"2023-10-03T17:34:42.272306+0000","last_became_peered":"2023-10-03T17:34:42.272306+0000","last_unstale":"2023-10-03T17:40:43.145263+0000","last_undegraded":"2023-10-03T17:40:43.145263+0000","last_fullsized":"2023-10-03T17:40:43.145263+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'20","last_scrub_stamp":"2023-10-03T17:40:10.708060+0000","last_deep_scrub":"46'20","last_deep_scrub_stamp":"2023-10-03T17:40:10.708060+0000","last_clean_scrub_stamp":"2023-10-03T17:40:10.708060+0000","objects_scrubbed":7,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:03:09.097252+0000","scrub_duration":0.042279491000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":200,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":309,"num_read_kb":242,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":301,"num_omap_keys":10,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"46'38623","reported_seq":75525,"reported_epoch":46,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T17:40:48.200370+0000","last_change":"2023-10-03T17:40:29.562114+0000","last_active":"2023-10-03T17:40:48.200370+0000","last_peered":"2023-10-03T17:40:48.200370+0000","last_clean":"2023-10-03T17:40:48.200370+0000","last_became_active":"2023-10-03T17:34:48.310612+0000","last_became_peered":"2023-10-03T17:34:48.310612+0000","last_unstale":"2023-10-03T17:40:48.200370+0000","last_undegraded":"2023-10-03T17:40:48.200370+0000","last_fullsized":"2023-10-03T17:40:48.200370+0000","mapping_epoch":16,"log_start":"46'28600","ondisk_log_start":"46'28600","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_clean_scrub_stamp":"2023-10-03T17:34:47.281095+0000","objects_scrubbed":155,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 18s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022375120000000001,"stat_sum":{"num_bytes":5214801939,"num_objects":1288,"num_object_clones":0,"num_object_copies":2576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1288,"num_whiteouts":0,"num_read":36813,"num_read_kb":132312,"num_write":38614,"num_write_kb":154450,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"46'8","reported_seq":75,"reported_epoch":46,"state":"active+clean","last_fresh":"2023-10-03T17:40:24.472668+0000","last_change":"2023-10-03T17:40:24.472734+0000","last_active":"2023-10-03T17:40:24.472668+0000","last_peered":"2023-10-03T17:40:24.472668+0000","last_clean":"2023-10-03T17:40:24.472668+0000","last_became_active":"2023-10-03T17:34:42.270194+0000","last_became_peered":"2023-10-03T17:34:42.270194+0000","last_unstale":"2023-10-03T17:40:24.472668+0000","last_undegraded":"2023-10-03T17:40:24.472668+0000","last_fullsized":"2023-10-03T17:40:24.472668+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'8","last_scrub_stamp":"2023-10-03T17:40:24.472573+0000","last_deep_scrub":"46'8","last_deep_scrub_stamp":"2023-10-03T17:40:24.472573+0000","last_clean_scrub_stamp":"2023-10-03T17:40:24.472573+0000","objects_scrubbed":4,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T18:28:04.735033+0000","scrub_duration":0.024990637999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":132,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":8,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"46'99","reported_seq":186,"reported_epoch":46,"state":"active+clean","last_fresh":"2023-10-03T17:40:10.452341+0000","last_change":"2023-10-03T17:40:10.452341+0000","last_active":"2023-10-03T17:40:10.452341+0000","last_peered":"2023-10-03T17:40:10.452341+0000","last_clean":"2023-10-03T17:40:10.452341+0000","last_became_active":"2023-10-03T17:34:39.502393+0000","last_became_peered":"2023-10-03T17:34:39.502393+0000","last_unstale":"2023-10-03T17:40:10.452341+0000","last_undegraded":"2023-10-03T17:40:10.452341+0000","last_fullsized":"2023-10-03T17:40:10.452341+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'99","last_scrub_stamp":"2023-10-03T17:40:10.449099+0000","last_deep_scrub":"46'99","last_deep_scrub_stamp":"2023-10-03T17:40:10.449099+0000","last_clean_scrub_stamp":"2023-10-03T17:40:10.449099+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T20:18:04.958666+0000","scrub_duration":0.043225012,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":20729217043,"num_objects":5121,"num_object_clones":0,"num_object_copies":10242,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5121,"num_whiteouts":0,"num_read":146678,"num_read_kb":526920,"num_write":151957,"num_write_kb":607822,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1244725248,"data_stored":1244717094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40098,"ondisk_log_size":40098,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":1328,"num_objects":63,"num_object_clones":0,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63,"num_whiteouts":0,"num_read":6261,"num_read_kb":5277,"num_write":203,"num_write_kb":203,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":2656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4150,"ondisk_log_size":4150,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":7460,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738448,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1078940,"kb_used_data":606484,"kb_used_omap":1,"kb_used_meta":472446,"kb_avail":93292900,"statfs":{"total":96636764160,"available":95531929600,"internally_reserved":0,"allocated":621039616,"data_stored":620752412,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":483785158},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,2,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 17:40:47 2023","interfaces":[{"interface":"back","average":{"1min":0.27900000000000003,"5min":0.40300000000000002,"15min":0.47799999999999998},"min":{"1min":0.17000000000000001,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":0.48799999999999999,"5min":4.4640000000000004,"15min":4.4640000000000004},"last":0.312},{"interface":"front","average":{"1min":0.378,"5min":0.38100000000000001,"15min":0.498},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":1.9850000000000001,"5min":1.9850000000000001,"15min":1.9850000000000001},"last":0.33100000000000002}]},{"osd":1,"last update":"Tue Oct 3 17:40:47 2023","interfaces":[{"interface":"back","average":{"1min":0.28999999999999998,"5min":0.41399999999999998,"15min":0.50800000000000001},"min":{"1min":0.189,"5min":0.16600000000000001,"15min":0.16600000000000001},"max":{"1min":0.41499999999999998,"5min":4.4260000000000002,"15min":4.4260000000000002},"last":0.26700000000000002},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.378,"15min":0.52700000000000002},"min":{"1min":0.20799999999999999,"5min":0.192,"15min":0.192},"max":{"1min":2.0579999999999998,"5min":2.0579999999999998,"15min":2.0579999999999998},"last":0.28399999999999997}]}]},{"osd":1,"up_from":8,"seq":34359738447,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":526060,"kb_used_data":300644,"kb_used_omap":1,"kb_used_meta":225406,"kb_avail":93845780,"statfs":{"total":96636764160,"available":96098078720,"internally_reserved":0,"allocated":307859456,"data_stored":307995563,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":230816198},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,2,8,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 17:39:56 2023","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.39200000000000002,"15min":0.47799999999999998},"min":{"1min":0.187,"5min":0.161,"15min":0.161},"max":{"1min":1.577,"5min":4.1879999999999997,"15min":4.1879999999999997},"last":0.255},{"interface":"front","average":{"1min":0.371,"5min":0.38700000000000001,"15min":0.48799999999999999},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":1.3899999999999999,"5min":1.532,"15min":1.532},"last":0.34499999999999997}]},{"osd":2,"last update":"Tue Oct 3 17:39:56 2023","interfaces":[{"interface":"back","average":{"1min":0.37,"5min":0.38700000000000001,"15min":0.49099999999999999},"min":{"1min":0.17799999999999999,"5min":0.159,"15min":0.159},"max":{"1min":2.4889999999999999,"5min":2.4889999999999999,"15min":2.4889999999999999},"last":0.215},{"interface":"front","average":{"1min":0.39600000000000002,"5min":0.39700000000000002,"15min":0.51900000000000002},"min":{"1min":0.19400000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":2.5219999999999998,"5min":2.5219999999999998,"15min":2.5219999999999998},"last":0.29499999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738447,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":518788,"kb_used_data":307260,"kb_used_omap":1,"kb_used_meta":211518,"kb_avail":93853052,"statfs":{"total":96636764160,"available":96105525248,"internally_reserved":0,"allocated":314634240,"data_stored":314827223,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":216594886},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,41,4,7,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 17:40:46 2023","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.39000000000000001,"15min":0.42599999999999999},"min":{"1min":0.154,"5min":0.154,"15min":0.154},"max":{"1min":5.593,"5min":5.593,"15min":5.593},"last":0.27600000000000002},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.41299999999999998,"15min":0.45700000000000002},"min":{"1min":0.193,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":3.1509999999999998,"5min":3.1509999999999998,"15min":3.1509999999999998},"last":0.39700000000000002}]},{"osd":2,"last update":"Tue Oct 3 17:40:46 2023","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.42799999999999999,"15min":0.53200000000000003},"min":{"1min":0.20699999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":1.4419999999999999,"5min":2.823,"15min":2.823},"last":0.313},{"interface":"front","average":{"1min":0.378,"5min":0.434,"15min":0.51700000000000002},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":1.421,"5min":2.9239999999999999,"15min":2.9239999999999999},"last":0.41499999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":879,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":314658816,"data_stored":314658816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":307769344,"data_stored":307765267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":622297088,"data_stored":622293011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-03T17:40:51.931 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-03T17:34:47.281095+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=34, tm_sec=47, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=40, tm_sec=6, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T17:40:51.931 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-03T17:34:47.281095+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=34, tm_sec=47, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=40, tm_sec=6, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T17:40:51.932 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-03T17:34:47.281095+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=34, tm_sec=47, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=40, tm_sec=6, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T17:40:51.932 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-03T17:34:47.281095+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=34, tm_sec=47, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=40, tm_sec=6, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T17:40:51.932 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T17:41:11.933 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T17:41:12.445 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-03T17:41:12.446 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-10-03T17:41:12.482 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":244,"stamp":"2023-10-03T17:41:12.043333+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20781751126,"num_objects":7421,"num_object_clones":2215,"num_object_copies":14842,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7421,"num_whiteouts":0,"num_read":164178,"num_read_kb":568111,"num_write":160933,"num_write_kb":644223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44698,"ondisk_log_size":44698,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2405344,"kb_used_data":1283588,"kb_used_omap":4,"kb_used_meta":1121723,"kb_avail":280710176,"statfs":{"total":289910292480,"available":287447220224,"internally_reserved":0,"allocated":1314394112,"data_stored":1825540210,"data_compressed":7460,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4782,"internal_metadata":1148644690},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,10,10,11,21,19],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":33362931,"num_objects":2235,"num_object_clones":2215,"num_object_copies":4470,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2235,"num_whiteouts":0,"num_read":6487,"num_read_kb":17655,"num_write":4035,"num_write_kb":15900,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.838919"},"pg_stats":[{"pgid":"2.7","version":"48'831","reported_seq":1453,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-10-03T17:41:08.057804+0000","last_change":"2023-10-03T17:40:17.407011+0000","last_active":"2023-10-03T17:41:08.057804+0000","last_peered":"2023-10-03T17:41:08.057804+0000","last_clean":"2023-10-03T17:41:08.057804+0000","last_became_active":"2023-10-03T17:34:42.276624+0000","last_became_peered":"2023-10-03T17:34:42.276624+0000","last_unstale":"2023-10-03T17:41:08.057804+0000","last_undegraded":"2023-10-03T17:41:08.057804+0000","last_fullsized":"2023-10-03T17:41:08.057804+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'673","last_scrub_stamp":"2023-10-03T17:40:17.406813+0000","last_deep_scrub":"46'673","last_deep_scrub_stamp":"2023-10-03T17:40:17.406813+0000","last_clean_scrub_stamp":"2023-10-03T17:40:17.406813+0000","objects_scrubbed":7,"log_size":831,"log_dups_size":0,"ondisk_log_size":831,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T21:38:35.056889+0000","scrub_duration":0.013778076,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":385,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":1185,"num_read_kb":990,"num_write":30,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":903,"num_omap_keys":30,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'1402","reported_seq":2757,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-10-03T17:41:08.063592+0000","last_change":"2023-10-03T17:40:13.376519+0000","last_active":"2023-10-03T17:41:08.063592+0000","last_peered":"2023-10-03T17:41:08.063592+0000","last_clean":"2023-10-03T17:41:08.063592+0000","last_became_active":"2023-10-03T17:34:42.271201+0000","last_became_peered":"2023-10-03T17:34:42.271201+0000","last_unstale":"2023-10-03T17:41:08.063592+0000","last_undegraded":"2023-10-03T17:41:08.063592+0000","last_fullsized":"2023-10-03T17:41:08.063592+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'1129","last_scrub_stamp":"2023-10-03T17:40:13.376332+0000","last_deep_scrub":"46'1129","last_deep_scrub_stamp":"2023-10-03T17:40:13.376332+0000","last_clean_scrub_stamp":"2023-10-03T17:40:13.376332+0000","objects_scrubbed":11,"log_size":1402,"log_dups_size":0,"ondisk_log_size":1402,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T18:59:17.088585+0000","scrub_duration":0.015998675,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":2550,"num_read_kb":2241,"num_write":86,"num_write_kb":86,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2912,"num_omap_keys":93,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"48'15","reported_seq":92,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-10-03T17:41:01.210661+0000","last_change":"2023-10-03T17:40:25.496010+0000","last_active":"2023-10-03T17:41:01.210661+0000","last_peered":"2023-10-03T17:41:01.210661+0000","last_clean":"2023-10-03T17:41:01.210661+0000","last_became_active":"2023-10-03T17:34:42.276941+0000","last_became_peered":"2023-10-03T17:34:42.276941+0000","last_unstale":"2023-10-03T17:41:01.210661+0000","last_undegraded":"2023-10-03T17:41:01.210661+0000","last_fullsized":"2023-10-03T17:41:01.210661+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'14","last_scrub_stamp":"2023-10-03T17:40:25.495827+0000","last_deep_scrub":"46'14","last_deep_scrub_stamp":"2023-10-03T17:40:25.495827+0000","last_clean_scrub_stamp":"2023-10-03T17:40:25.495827+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:05:44.939663+0000","scrub_duration":0.0133078,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":300,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'1125","reported_seq":2143,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-10-03T17:41:08.065652+0000","last_change":"2023-10-03T17:40:26.513902+0000","last_active":"2023-10-03T17:41:08.065652+0000","last_peered":"2023-10-03T17:41:08.065652+0000","last_clean":"2023-10-03T17:41:08.065652+0000","last_became_active":"2023-10-03T17:34:42.276459+0000","last_became_peered":"2023-10-03T17:34:42.276459+0000","last_unstale":"2023-10-03T17:41:08.065652+0000","last_undegraded":"2023-10-03T17:41:08.065652+0000","last_fullsized":"2023-10-03T17:41:08.065652+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'964","last_scrub_stamp":"2023-10-03T17:40:26.513729+0000","last_deep_scrub":"46'964","last_deep_scrub_stamp":"2023-10-03T17:40:26.513729+0000","last_clean_scrub_stamp":"2023-10-03T17:40:26.513729+0000","objects_scrubbed":10,"log_size":1125,"log_dups_size":0,"ondisk_log_size":1125,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:41:33.182751+0000","scrub_duration":0.018190280999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1820,"num_read_kb":1534,"num_write":54,"num_write_kb":54,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1731,"num_omap_keys":80,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"48'41934","reported_seq":81471,"reported_epoch":48,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T17:41:08.311324+0000","last_change":"2023-10-03T17:40:11.716583+0000","last_active":"2023-10-03T17:41:08.311324+0000","last_peered":"2023-10-03T17:41:08.311324+0000","last_clean":"2023-10-03T17:41:08.311324+0000","last_became_active":"2023-10-03T17:34:48.310701+0000","last_became_peered":"2023-10-03T17:34:48.310701+0000","last_unstale":"2023-10-03T17:41:08.311324+0000","last_undegraded":"2023-10-03T17:41:08.311324+0000","last_fullsized":"2023-10-03T17:41:08.311324+0000","mapping_epoch":16,"log_start":"46'31905","ondisk_log_start":"46'31905","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_clean_scrub_stamp":"2023-10-03T17:34:47.281095+0000","objects_scrubbed":446,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 56s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040768800000000002,"stat_sum":{"num_bytes":5352210432,"num_objects":1902,"num_object_clones":582,"num_object_copies":3804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1902,"num_whiteouts":0,"num_read":39986,"num_read_kb":144272,"num_write":41320,"num_write_kb":165280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"48'1052","reported_seq":1875,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-10-03T17:41:08.145354+0000","last_change":"2023-10-03T17:40:08.675119+0000","last_active":"2023-10-03T17:41:08.145354+0000","last_peered":"2023-10-03T17:41:08.145354+0000","last_clean":"2023-10-03T17:41:08.145354+0000","last_became_active":"2023-10-03T17:34:42.270625+0000","last_became_peered":"2023-10-03T17:34:42.270625+0000","last_unstale":"2023-10-03T17:41:08.145354+0000","last_undegraded":"2023-10-03T17:41:08.145354+0000","last_fullsized":"2023-10-03T17:41:08.145354+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'841","last_scrub_stamp":"2023-10-03T17:40:08.674910+0000","last_deep_scrub":"46'841","last_deep_scrub_stamp":"2023-10-03T17:40:08.674910+0000","last_clean_scrub_stamp":"2023-10-03T17:40:08.674910+0000","objects_scrubbed":11,"log_size":1052,"log_dups_size":0,"ondisk_log_size":1052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T21:05:42.790614+0000","scrub_duration":0.039189555000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":352,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":1593,"num_read_kb":1335,"num_write":57,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2564,"num_omap_keys":60,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"48'39214","reported_seq":76552,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-10-03T17:41:08.293835+0000","last_change":"2023-10-03T17:40:55.606282+0000","last_active":"2023-10-03T17:41:08.293835+0000","last_peered":"2023-10-03T17:41:08.293835+0000","last_clean":"2023-10-03T17:41:08.293835+0000","last_became_active":"2023-10-03T17:34:48.310555+0000","last_became_peered":"2023-10-03T17:34:48.310555+0000","last_unstale":"2023-10-03T17:41:08.293835+0000","last_undegraded":"2023-10-03T17:41:08.293835+0000","last_fullsized":"2023-10-03T17:41:08.293835+0000","mapping_epoch":16,"log_start":"46'29207","ondisk_log_start":"46'29207","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_clean_scrub_stamp":"2023-10-03T17:34:47.281095+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086289699999999995,"stat_sum":{"num_bytes":5044051968,"num_objects":1806,"num_object_clones":564,"num_object_copies":3612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1806,"num_whiteouts":0,"num_read":37822,"num_read_kb":135920,"num_write":38650,"num_write_kb":154600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"48'44","reported_seq":656,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-10-03T17:41:05.206564+0000","last_change":"2023-10-03T17:40:17.641493+0000","last_active":"2023-10-03T17:41:05.206564+0000","last_peered":"2023-10-03T17:41:05.206564+0000","last_clean":"2023-10-03T17:41:05.206564+0000","last_became_active":"2023-10-03T17:34:42.278196+0000","last_became_peered":"2023-10-03T17:34:42.278196+0000","last_unstale":"2023-10-03T17:41:05.206564+0000","last_undegraded":"2023-10-03T17:41:05.206564+0000","last_fullsized":"2023-10-03T17:41:05.206564+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'30","last_scrub_stamp":"2023-10-03T17:40:17.640969+0000","last_deep_scrub":"46'30","last_deep_scrub_stamp":"2023-10-03T17:40:17.640969+0000","last_clean_scrub_stamp":"2023-10-03T17:40:17.640969+0000","objects_scrubbed":6,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:04:02.954896+0000","scrub_duration":0.071054565,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":218,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1171,"num_read_kb":985,"num_write":26,"num_write_kb":26,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":903,"num_omap_keys":30,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"48'40325","reported_seq":78440,"reported_epoch":48,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T17:41:08.312610+0000","last_change":"2023-10-03T17:40:10.712199+0000","last_active":"2023-10-03T17:41:08.312610+0000","last_peered":"2023-10-03T17:41:08.312610+0000","last_clean":"2023-10-03T17:41:08.312610+0000","last_became_active":"2023-10-03T17:34:48.310912+0000","last_became_peered":"2023-10-03T17:34:48.310912+0000","last_unstale":"2023-10-03T17:41:08.312610+0000","last_undegraded":"2023-10-03T17:41:08.312610+0000","last_fullsized":"2023-10-03T17:41:08.312610+0000","mapping_epoch":16,"log_start":"46'30304","ondisk_log_start":"46'30304","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_clean_scrub_stamp":"2023-10-03T17:34:47.281095+0000","objects_scrubbed":583,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 57s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045661900000000001,"stat_sum":{"num_bytes":5159161856,"num_objects":1781,"num_object_clones":510,"num_object_copies":3562,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1781,"num_whiteouts":0,"num_read":38489,"num_read_kb":139624,"num_write":39778,"num_write_kb":159112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"48'28","reported_seq":289,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-10-03T17:41:04.851855+0000","last_change":"2023-10-03T17:40:10.708344+0000","last_active":"2023-10-03T17:41:04.851855+0000","last_peered":"2023-10-03T17:41:04.851855+0000","last_clean":"2023-10-03T17:41:04.851855+0000","last_became_active":"2023-10-03T17:34:42.272306+0000","last_became_peered":"2023-10-03T17:34:42.272306+0000","last_unstale":"2023-10-03T17:41:04.851855+0000","last_undegraded":"2023-10-03T17:41:04.851855+0000","last_fullsized":"2023-10-03T17:41:04.851855+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'20","last_scrub_stamp":"2023-10-03T17:40:10.708060+0000","last_deep_scrub":"46'20","last_deep_scrub_stamp":"2023-10-03T17:40:10.708060+0000","last_clean_scrub_stamp":"2023-10-03T17:40:10.708060+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:03:09.097252+0000","scrub_duration":0.042279491000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":401,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":425,"num_read_kb":339,"num_write":17,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":301,"num_omap_keys":10,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"48'41344","reported_seq":79867,"reported_epoch":48,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T17:41:08.311978+0000","last_change":"2023-10-03T17:40:29.562114+0000","last_active":"2023-10-03T17:41:08.311978+0000","last_peered":"2023-10-03T17:41:08.311978+0000","last_clean":"2023-10-03T17:41:08.311978+0000","last_became_active":"2023-10-03T17:34:48.310612+0000","last_became_peered":"2023-10-03T17:34:48.310612+0000","last_unstale":"2023-10-03T17:41:08.311978+0000","last_undegraded":"2023-10-03T17:41:08.311978+0000","last_fullsized":"2023-10-03T17:41:08.311978+0000","mapping_epoch":16,"log_start":"46'31308","ondisk_log_start":"46'31308","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_clean_scrub_stamp":"2023-10-03T17:34:47.281095+0000","objects_scrubbed":302,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 38s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022375120000000001,"stat_sum":{"num_bytes":5225734163,"num_objects":1847,"num_object_clones":559,"num_object_copies":3694,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1847,"num_whiteouts":0,"num_read":38966,"num_read_kb":140760,"num_write":40765,"num_write_kb":163054,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"47'9","reported_seq":79,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-10-03T17:41:00.130118+0000","last_change":"2023-10-03T17:40:24.472734+0000","last_active":"2023-10-03T17:41:00.130118+0000","last_peered":"2023-10-03T17:41:00.130118+0000","last_clean":"2023-10-03T17:41:00.130118+0000","last_became_active":"2023-10-03T17:34:42.270194+0000","last_became_peered":"2023-10-03T17:34:42.270194+0000","last_unstale":"2023-10-03T17:41:00.130118+0000","last_undegraded":"2023-10-03T17:41:00.130118+0000","last_fullsized":"2023-10-03T17:41:00.130118+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'8","last_scrub_stamp":"2023-10-03T17:40:24.472573+0000","last_deep_scrub":"46'8","last_deep_scrub_stamp":"2023-10-03T17:40:24.472573+0000","last_clean_scrub_stamp":"2023-10-03T17:40:24.472573+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T18:28:04.735033+0000","scrub_duration":0.024990637999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":182,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":8,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"46'99","reported_seq":188,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-10-03T17:41:00.112863+0000","last_change":"2023-10-03T17:40:10.452341+0000","last_active":"2023-10-03T17:41:00.112863+0000","last_peered":"2023-10-03T17:41:00.112863+0000","last_clean":"2023-10-03T17:41:00.112863+0000","last_became_active":"2023-10-03T17:34:39.502393+0000","last_became_peered":"2023-10-03T17:34:39.502393+0000","last_unstale":"2023-10-03T17:41:00.112863+0000","last_undegraded":"2023-10-03T17:41:00.112863+0000","last_fullsized":"2023-10-03T17:41:00.112863+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'99","last_scrub_stamp":"2023-10-03T17:40:10.449099+0000","last_deep_scrub":"46'99","last_deep_scrub_stamp":"2023-10-03T17:40:10.449099+0000","last_clean_scrub_stamp":"2023-10-03T17:40:10.449099+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T20:18:04.958666+0000","scrub_duration":0.043225012,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":20781158419,"num_objects":7336,"num_object_clones":2215,"num_object_copies":14672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7336,"num_whiteouts":0,"num_read":155263,"num_read_kb":560576,"num_write":160513,"num_write_kb":642046,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1314992128,"data_stored":1872408614,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40093,"ondisk_log_size":40093,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2339,"num_objects":83,"num_object_clones":0,"num_object_copies":166,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83,"num_whiteouts":0,"num_read":8809,"num_read_kb":7444,"num_write":283,"num_write_kb":283,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":4678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4506,"ondisk_log_size":4506,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":7460,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738453,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1181524,"kb_used_data":641100,"kb_used_omap":1,"kb_used_meta":540414,"kb_avail":93190316,"statfs":{"total":96636764160,"available":95426883584,"internally_reserved":0,"allocated":656486400,"data_stored":906198646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":553384390},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,2,3,9,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 17:40:47 2023","interfaces":[{"interface":"back","average":{"1min":0.27900000000000003,"5min":0.40300000000000002,"15min":0.47799999999999998},"min":{"1min":0.17000000000000001,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":0.48799999999999999,"5min":4.4640000000000004,"15min":4.4640000000000004},"last":0.25800000000000001},{"interface":"front","average":{"1min":0.378,"5min":0.38100000000000001,"15min":0.498},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":1.9850000000000001,"5min":1.9850000000000001,"15min":1.9850000000000001},"last":0.34499999999999997}]},{"osd":1,"last update":"Tue Oct 3 17:40:47 2023","interfaces":[{"interface":"back","average":{"1min":0.28999999999999998,"5min":0.41399999999999998,"15min":0.50800000000000001},"min":{"1min":0.189,"5min":0.16600000000000001,"15min":0.16600000000000001},"max":{"1min":0.41499999999999998,"5min":4.4260000000000002,"15min":4.4260000000000002},"last":0.30399999999999999},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.378,"15min":0.52700000000000002},"min":{"1min":0.20799999999999999,"5min":0.192,"15min":0.192},"max":{"1min":2.0579999999999998,"5min":2.0579999999999998,"15min":2.0579999999999998},"last":0.48299999999999998}]}]},{"osd":1,"up_from":8,"seq":34359738451,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":615360,"kb_used_data":317752,"kb_used_omap":1,"kb_used_meta":297598,"kb_avail":93756480,"statfs":{"total":96636764160,"available":96006635520,"internally_reserved":0,"allocated":325378048,"data_stored":455091864,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":304740806},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,5,3,5,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 17:40:59 2023","interfaces":[{"interface":"back","average":{"1min":0.32400000000000001,"5min":0.35299999999999998,"15min":0.46500000000000002},"min":{"1min":0.17100000000000001,"5min":0.161,"15min":0.161},"max":{"1min":1.8120000000000001,"5min":4.1879999999999997,"15min":4.1879999999999997},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.30599999999999999,"5min":0.34000000000000002,"15min":0.47199999999999998},"min":{"1min":0.17399999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.94099999999999995,"5min":1.532,"15min":1.532},"last":0.20999999999999999}]},{"osd":2,"last update":"Tue Oct 3 17:40:59 2023","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.373,"15min":0.48599999999999999},"min":{"1min":0.20499999999999999,"5min":0.159,"15min":0.159},"max":{"1min":3.0390000000000001,"5min":3.0390000000000001,"15min":3.0390000000000001},"last":0.30599999999999999},{"interface":"front","average":{"1min":0.46800000000000003,"5min":0.375,"15min":0.51200000000000001},"min":{"1min":0.23400000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":3.0699999999999998,"5min":3.0699999999999998,"15min":3.0699999999999998},"last":0.32800000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738451,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":608460,"kb_used_data":324736,"kb_used_omap":1,"kb_used_meta":283710,"kb_avail":93763380,"statfs":{"total":96636764160,"available":96013701120,"internally_reserved":0,"allocated":332529664,"data_stored":464249700,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":290519494},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,3,5,7,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 17:40:46 2023","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.39000000000000001,"15min":0.42599999999999999},"min":{"1min":0.154,"5min":0.154,"15min":0.154},"max":{"1min":5.593,"5min":5.593,"15min":5.593},"last":0.189},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.41299999999999998,"15min":0.45700000000000002},"min":{"1min":0.193,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":3.1509999999999998,"5min":3.1509999999999998,"15min":3.1509999999999998},"last":0.23699999999999999}]},{"osd":2,"last update":"Tue Oct 3 17:40:46 2023","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.42799999999999999,"15min":0.53200000000000003},"min":{"1min":0.20699999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":1.4419999999999999,"5min":2.823,"15min":2.823},"last":0.20699999999999999},{"interface":"front","average":{"1min":0.378,"5min":0.434,"15min":0.51700000000000002},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":1.421,"5min":2.9239999999999999,"15min":2.9239999999999999},"last":0.30099999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":245760,"data_stored":2339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":332181504,"data_stored":469364736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":325308416,"data_stored":466866195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":657502208,"data_stored":936177683,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-03T17:41:12.484 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-03T17:34:47.281095+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=34, tm_sec=47, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=40, tm_sec=6, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T17:41:12.484 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-03T17:34:47.281095+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=34, tm_sec=47, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=40, tm_sec=6, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T17:41:12.485 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-03T17:34:47.281095+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=34, tm_sec=47, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=40, tm_sec=6, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T17:41:12.485 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-03T17:34:47.281095+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=34, tm_sec=47, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=40, tm_sec=6, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T17:41:12.485 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T17:41:25.328 INFO:tasks.ceph.osd.0.smithi111.stderr:2023-10-03T17:41:25.326+0000 7ffbac289640 -1 reset not still connected to 0x5563a7ff65b0 2023-10-03T17:41:25.329 INFO:tasks.ceph.osd.0.smithi111.stderr:2023-10-03T17:41:25.326+0000 7ffbac289640 -1 reset not still connected to 0x5563a7ff68f0 2023-10-03T17:41:25.329 INFO:tasks.ceph.osd.0.smithi111.stderr:2023-10-03T17:41:25.326+0000 7ffbac289640 -1 reset not still connected to 0x5563a838e1a0 2023-10-03T17:41:25.329 INFO:tasks.ceph.osd.0.smithi111.stderr:2023-10-03T17:41:25.326+0000 7ffbac289640 -1 reset not still connected to 0x5563a851aea0 2023-10-03T17:41:32.486 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T17:41:33.092 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-03T17:41:33.094 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-10-03T17:41:33.118 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":254,"stamp":"2023-10-03T17:41:32.046474+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20855450454,"num_objects":9852,"num_object_clones":4646,"num_object_copies":19704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9852,"num_whiteouts":0,"num_read":172990,"num_read_kb":602675,"num_write":169349,"num_write_kb":677887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":45107,"ondisk_log_size":45107,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2545868,"kb_used_data":1351920,"kb_used_omap":4,"kb_used_meta":1193915,"kb_avail":280569652,"statfs":{"total":289910292480,"available":287303323648,"internally_reserved":0,"allocated":1384366080,"data_stored":2551498866,"data_compressed":7460,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4782,"internal_metadata":1222569298},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,2,3,10,5,8,11,7],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":32501760,"num_objects":747,"num_object_clones":747,"num_object_copies":1494,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":747,"num_whiteouts":0,"num_read":4604,"num_read_kb":18052,"num_write":4288,"num_write_kb":17152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001897"},"pg_stats":[{"pgid":"2.7","version":"48'885","reported_seq":1520,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-10-03T17:41:28.267727+0000","last_change":"2023-10-03T17:40:17.407011+0000","last_active":"2023-10-03T17:41:28.267727+0000","last_peered":"2023-10-03T17:41:28.267727+0000","last_clean":"2023-10-03T17:41:28.267727+0000","last_became_active":"2023-10-03T17:34:42.276624+0000","last_became_peered":"2023-10-03T17:34:42.276624+0000","last_unstale":"2023-10-03T17:41:28.267727+0000","last_undegraded":"2023-10-03T17:41:28.267727+0000","last_fullsized":"2023-10-03T17:41:28.267727+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'673","last_scrub_stamp":"2023-10-03T17:40:17.406813+0000","last_deep_scrub":"46'673","last_deep_scrub_stamp":"2023-10-03T17:40:17.406813+0000","last_clean_scrub_stamp":"2023-10-03T17:40:17.406813+0000","objects_scrubbed":7,"log_size":885,"log_dups_size":0,"ondisk_log_size":885,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T21:38:35.056889+0000","scrub_duration":0.013778076,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":385,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":1185,"num_read_kb":990,"num_write":30,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":903,"num_omap_keys":30,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'1492","reported_seq":2868,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-10-03T17:41:28.270248+0000","last_change":"2023-10-03T17:40:13.376519+0000","last_active":"2023-10-03T17:41:28.270248+0000","last_peered":"2023-10-03T17:41:28.270248+0000","last_clean":"2023-10-03T17:41:28.270248+0000","last_became_active":"2023-10-03T17:34:42.271201+0000","last_became_peered":"2023-10-03T17:34:42.271201+0000","last_unstale":"2023-10-03T17:41:28.270248+0000","last_undegraded":"2023-10-03T17:41:28.270248+0000","last_fullsized":"2023-10-03T17:41:28.270248+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'1129","last_scrub_stamp":"2023-10-03T17:40:13.376332+0000","last_deep_scrub":"46'1129","last_deep_scrub_stamp":"2023-10-03T17:40:13.376332+0000","last_clean_scrub_stamp":"2023-10-03T17:40:13.376332+0000","objects_scrubbed":11,"log_size":1492,"log_dups_size":0,"ondisk_log_size":1492,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T18:59:17.088585+0000","scrub_duration":0.015998675,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":2550,"num_read_kb":2241,"num_write":86,"num_write_kb":86,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2912,"num_omap_keys":93,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"48'15","reported_seq":92,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-10-03T17:41:01.210661+0000","last_change":"2023-10-03T17:40:25.496010+0000","last_active":"2023-10-03T17:41:01.210661+0000","last_peered":"2023-10-03T17:41:01.210661+0000","last_clean":"2023-10-03T17:41:01.210661+0000","last_became_active":"2023-10-03T17:34:42.276941+0000","last_became_peered":"2023-10-03T17:34:42.276941+0000","last_unstale":"2023-10-03T17:41:01.210661+0000","last_undegraded":"2023-10-03T17:41:01.210661+0000","last_fullsized":"2023-10-03T17:41:01.210661+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'14","last_scrub_stamp":"2023-10-03T17:40:25.495827+0000","last_deep_scrub":"46'14","last_deep_scrub_stamp":"2023-10-03T17:40:25.495827+0000","last_clean_scrub_stamp":"2023-10-03T17:40:25.495827+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:05:44.939663+0000","scrub_duration":0.0133078,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":300,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'1197","reported_seq":2231,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-10-03T17:41:28.268246+0000","last_change":"2023-10-03T17:40:26.513902+0000","last_active":"2023-10-03T17:41:28.268246+0000","last_peered":"2023-10-03T17:41:28.268246+0000","last_clean":"2023-10-03T17:41:28.268246+0000","last_became_active":"2023-10-03T17:34:42.276459+0000","last_became_peered":"2023-10-03T17:34:42.276459+0000","last_unstale":"2023-10-03T17:41:28.268246+0000","last_undegraded":"2023-10-03T17:41:28.268246+0000","last_fullsized":"2023-10-03T17:41:28.268246+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'964","last_scrub_stamp":"2023-10-03T17:40:26.513729+0000","last_deep_scrub":"46'964","last_deep_scrub_stamp":"2023-10-03T17:40:26.513729+0000","last_clean_scrub_stamp":"2023-10-03T17:40:26.513729+0000","objects_scrubbed":10,"log_size":1197,"log_dups_size":0,"ondisk_log_size":1197,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:41:33.182751+0000","scrub_duration":0.018190280999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1820,"num_read_kb":1534,"num_write":54,"num_write_kb":54,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1731,"num_omap_keys":80,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"48'44755","reported_seq":85840,"reported_epoch":48,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T17:41:28.423787+0000","last_change":"2023-10-03T17:40:11.716583+0000","last_active":"2023-10-03T17:41:28.423787+0000","last_peered":"2023-10-03T17:41:28.423787+0000","last_clean":"2023-10-03T17:41:28.423787+0000","last_became_active":"2023-10-03T17:34:48.310701+0000","last_became_peered":"2023-10-03T17:34:48.310701+0000","last_unstale":"2023-10-03T17:41:28.423787+0000","last_undegraded":"2023-10-03T17:41:28.423787+0000","last_fullsized":"2023-10-03T17:41:28.423787+0000","mapping_epoch":16,"log_start":"46'34716","ondisk_log_start":"46'34716","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_clean_scrub_stamp":"2023-10-03T17:34:47.281095+0000","objects_scrubbed":677,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 76s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040768800000000002,"stat_sum":{"num_bytes":5371031552,"num_objects":2523,"num_object_clones":1203,"num_object_copies":5046,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2523,"num_whiteouts":0,"num_read":42138,"num_read_kb":152720,"num_write":43475,"num_write_kb":173900,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"48'1108","reported_seq":1947,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-10-03T17:41:27.460809+0000","last_change":"2023-10-03T17:40:08.675119+0000","last_active":"2023-10-03T17:41:27.460809+0000","last_peered":"2023-10-03T17:41:27.460809+0000","last_clean":"2023-10-03T17:41:27.460809+0000","last_became_active":"2023-10-03T17:34:42.270625+0000","last_became_peered":"2023-10-03T17:34:42.270625+0000","last_unstale":"2023-10-03T17:41:27.460809+0000","last_undegraded":"2023-10-03T17:41:27.460809+0000","last_fullsized":"2023-10-03T17:41:27.460809+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'841","last_scrub_stamp":"2023-10-03T17:40:08.674910+0000","last_deep_scrub":"46'841","last_deep_scrub_stamp":"2023-10-03T17:40:08.674910+0000","last_clean_scrub_stamp":"2023-10-03T17:40:08.674910+0000","objects_scrubbed":11,"log_size":1108,"log_dups_size":0,"ondisk_log_size":1108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T21:05:42.790614+0000","scrub_duration":0.039189555000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":352,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":1593,"num_read_kb":1335,"num_write":57,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2564,"num_omap_keys":60,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"48'41905","reported_seq":80838,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-10-03T17:41:28.483532+0000","last_change":"2023-10-03T17:41:27.569625+0000","last_active":"2023-10-03T17:41:28.483532+0000","last_peered":"2023-10-03T17:41:28.483532+0000","last_clean":"2023-10-03T17:41:28.483532+0000","last_became_active":"2023-10-03T17:34:48.310555+0000","last_became_peered":"2023-10-03T17:34:48.310555+0000","last_unstale":"2023-10-03T17:41:28.483532+0000","last_undegraded":"2023-10-03T17:41:28.483532+0000","last_fullsized":"2023-10-03T17:41:28.483532+0000","mapping_epoch":16,"log_start":"46'31812","ondisk_log_start":"46'31812","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_clean_scrub_stamp":"2023-10-03T17:34:47.281095+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086289699999999995,"stat_sum":{"num_bytes":5063983104,"num_objects":2378,"num_object_clones":1136,"num_object_copies":4756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2378,"num_whiteouts":0,"num_read":39981,"num_read_kb":144412,"num_write":40769,"num_write_kb":163076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"48'47","reported_seq":668,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-10-03T17:41:25.693594+0000","last_change":"2023-10-03T17:40:17.641493+0000","last_active":"2023-10-03T17:41:25.693594+0000","last_peered":"2023-10-03T17:41:25.693594+0000","last_clean":"2023-10-03T17:41:25.693594+0000","last_became_active":"2023-10-03T17:34:42.278196+0000","last_became_peered":"2023-10-03T17:34:42.278196+0000","last_unstale":"2023-10-03T17:41:25.693594+0000","last_undegraded":"2023-10-03T17:41:25.693594+0000","last_fullsized":"2023-10-03T17:41:25.693594+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'30","last_scrub_stamp":"2023-10-03T17:40:17.640969+0000","last_deep_scrub":"46'30","last_deep_scrub_stamp":"2023-10-03T17:40:17.640969+0000","last_clean_scrub_stamp":"2023-10-03T17:40:17.640969+0000","objects_scrubbed":6,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:04:02.954896+0000","scrub_duration":0.071054565,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":218,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1171,"num_read_kb":985,"num_write":26,"num_write_kb":26,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":903,"num_omap_keys":30,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"48'43054","reported_seq":82887,"reported_epoch":48,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T17:41:28.421495+0000","last_change":"2023-10-03T17:40:10.712199+0000","last_active":"2023-10-03T17:41:28.421495+0000","last_peered":"2023-10-03T17:41:28.421495+0000","last_clean":"2023-10-03T17:41:28.421495+0000","last_became_active":"2023-10-03T17:34:48.310912+0000","last_became_peered":"2023-10-03T17:34:48.310912+0000","last_unstale":"2023-10-03T17:41:28.421495+0000","last_undegraded":"2023-10-03T17:41:28.421495+0000","last_fullsized":"2023-10-03T17:41:28.421495+0000","mapping_epoch":16,"log_start":"46'33010","ondisk_log_start":"46'33010","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_clean_scrub_stamp":"2023-10-03T17:34:47.281095+0000","objects_scrubbed":741,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 77s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045661900000000001,"stat_sum":{"num_bytes":5176647680,"num_objects":2413,"num_object_clones":1142,"num_object_copies":4826,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2413,"num_whiteouts":0,"num_read":40813,"num_read_kb":148752,"num_write":41839,"num_write_kb":167356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"48'29","reported_seq":293,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-10-03T17:41:20.678044+0000","last_change":"2023-10-03T17:40:10.708344+0000","last_active":"2023-10-03T17:41:20.678044+0000","last_peered":"2023-10-03T17:41:20.678044+0000","last_clean":"2023-10-03T17:41:20.678044+0000","last_became_active":"2023-10-03T17:34:42.272306+0000","last_became_peered":"2023-10-03T17:34:42.272306+0000","last_unstale":"2023-10-03T17:41:20.678044+0000","last_undegraded":"2023-10-03T17:41:20.678044+0000","last_fullsized":"2023-10-03T17:41:20.678044+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'20","last_scrub_stamp":"2023-10-03T17:40:10.708060+0000","last_deep_scrub":"46'20","last_deep_scrub_stamp":"2023-10-03T17:40:10.708060+0000","last_clean_scrub_stamp":"2023-10-03T17:40:10.708060+0000","objects_scrubbed":7,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:03:09.097252+0000","scrub_duration":0.042279491000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":401,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":425,"num_read_kb":339,"num_write":17,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":301,"num_omap_keys":10,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"48'44070","reported_seq":84184,"reported_epoch":48,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T17:41:28.498649+0000","last_change":"2023-10-03T17:40:29.562114+0000","last_active":"2023-10-03T17:41:28.498649+0000","last_peered":"2023-10-03T17:41:28.498649+0000","last_clean":"2023-10-03T17:41:28.498649+0000","last_became_active":"2023-10-03T17:34:48.310612+0000","last_became_peered":"2023-10-03T17:34:48.310612+0000","last_unstale":"2023-10-03T17:41:28.498649+0000","last_undegraded":"2023-10-03T17:41:28.498649+0000","last_fullsized":"2023-10-03T17:41:28.498649+0000","mapping_epoch":16,"log_start":"46'34020","ondisk_log_start":"46'34020","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_clean_scrub_stamp":"2023-10-03T17:34:47.281095+0000","objects_scrubbed":489,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 58s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022375120000000001,"stat_sum":{"num_bytes":5243195411,"num_objects":2453,"num_object_clones":1165,"num_object_copies":4906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2453,"num_whiteouts":0,"num_read":41143,"num_read_kb":149256,"num_write":42846,"num_write_kb":171378,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"47'9","reported_seq":79,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-10-03T17:41:00.130118+0000","last_change":"2023-10-03T17:40:24.472734+0000","last_active":"2023-10-03T17:41:00.130118+0000","last_peered":"2023-10-03T17:41:00.130118+0000","last_clean":"2023-10-03T17:41:00.130118+0000","last_became_active":"2023-10-03T17:34:42.270194+0000","last_became_peered":"2023-10-03T17:34:42.270194+0000","last_unstale":"2023-10-03T17:41:00.130118+0000","last_undegraded":"2023-10-03T17:41:00.130118+0000","last_fullsized":"2023-10-03T17:41:00.130118+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'8","last_scrub_stamp":"2023-10-03T17:40:24.472573+0000","last_deep_scrub":"46'8","last_deep_scrub_stamp":"2023-10-03T17:40:24.472573+0000","last_clean_scrub_stamp":"2023-10-03T17:40:24.472573+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T18:28:04.735033+0000","scrub_duration":0.024990637999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":182,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":8,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"46'99","reported_seq":188,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-10-03T17:41:00.112863+0000","last_change":"2023-10-03T17:40:10.452341+0000","last_active":"2023-10-03T17:41:00.112863+0000","last_peered":"2023-10-03T17:41:00.112863+0000","last_clean":"2023-10-03T17:41:00.112863+0000","last_became_active":"2023-10-03T17:34:39.502393+0000","last_became_peered":"2023-10-03T17:34:39.502393+0000","last_unstale":"2023-10-03T17:41:00.112863+0000","last_undegraded":"2023-10-03T17:41:00.112863+0000","last_fullsized":"2023-10-03T17:41:00.112863+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'99","last_scrub_stamp":"2023-10-03T17:40:10.449099+0000","last_deep_scrub":"46'99","last_deep_scrub_stamp":"2023-10-03T17:40:10.449099+0000","last_clean_scrub_stamp":"2023-10-03T17:40:10.449099+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T20:18:04.958666+0000","scrub_duration":0.043225012,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":20854857747,"num_objects":9767,"num_object_clones":4646,"num_object_copies":19534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9767,"num_whiteouts":0,"num_read":164075,"num_read_kb":595140,"num_write":168929,"num_write_kb":675710,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1383772160,"data_stored":2555527206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40226,"ondisk_log_size":40226,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2339,"num_objects":83,"num_object_clones":0,"num_object_copies":166,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83,"num_whiteouts":0,"num_read":8809,"num_read_kb":7444,"num_write":283,"num_write_kb":283,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":4678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4782,"ondisk_log_size":4782,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":7460,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738457,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1248436,"kb_used_data":676072,"kb_used_omap":1,"kb_used_meta":572350,"kb_avail":93123404,"statfs":{"total":96636764160,"available":95358365696,"internally_reserved":0,"allocated":692297728,"data_stored":1277455990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":586086854},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,2,4,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 17:40:47 2023","interfaces":[{"interface":"back","average":{"1min":0.27900000000000003,"5min":0.40300000000000002,"15min":0.47799999999999998},"min":{"1min":0.17000000000000001,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":0.48799999999999999,"5min":4.4640000000000004,"15min":4.4640000000000004},"last":0.187},{"interface":"front","average":{"1min":0.378,"5min":0.38100000000000001,"15min":0.498},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":1.9850000000000001,"5min":1.9850000000000001,"15min":1.9850000000000001},"last":0.25600000000000001}]},{"osd":1,"last update":"Tue Oct 3 17:40:47 2023","interfaces":[{"interface":"back","average":{"1min":0.28999999999999998,"5min":0.41399999999999998,"15min":0.50800000000000001},"min":{"1min":0.189,"5min":0.16600000000000001,"15min":0.16600000000000001},"max":{"1min":0.41499999999999998,"5min":4.4260000000000002,"15min":4.4260000000000002},"last":0.23999999999999999},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.378,"15min":0.52700000000000002},"min":{"1min":0.20799999999999999,"5min":0.192,"15min":0.192},"max":{"1min":2.0579999999999998,"5min":2.0579999999999998,"15min":2.0579999999999998},"last":0.34599999999999997}]}]},{"osd":1,"up_from":8,"seq":34359738455,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":649960,"kb_used_data":334304,"kb_used_omap":1,"kb_used_meta":315646,"kb_avail":93721880,"statfs":{"total":96636764160,"available":95971205120,"internally_reserved":0,"allocated":342327296,"data_stored":630187672,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":323221958},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 17:40:59 2023","interfaces":[{"interface":"back","average":{"1min":0.32400000000000001,"5min":0.35299999999999998,"15min":0.46500000000000002},"min":{"1min":0.17100000000000001,"5min":0.161,"15min":0.161},"max":{"1min":1.8120000000000001,"5min":4.1879999999999997,"15min":4.1879999999999997},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.30599999999999999,"5min":0.34000000000000002,"15min":0.47199999999999998},"min":{"1min":0.17399999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.94099999999999995,"5min":1.532,"15min":1.532},"last":0.32100000000000001}]},{"osd":2,"last update":"Tue Oct 3 17:40:59 2023","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.373,"15min":0.48599999999999999},"min":{"1min":0.20499999999999999,"5min":0.159,"15min":0.159},"max":{"1min":3.0390000000000001,"5min":3.0390000000000001,"15min":3.0390000000000001},"last":0.30399999999999999},{"interface":"front","average":{"1min":0.46800000000000003,"5min":0.375,"15min":0.51200000000000001},"min":{"1min":0.23400000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":3.0699999999999998,"5min":3.0699999999999998,"15min":3.0699999999999998},"last":0.28599999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738455,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":647472,"kb_used_data":341544,"kb_used_omap":1,"kb_used_meta":305918,"kb_avail":93724368,"statfs":{"total":96636764160,"available":95973752832,"internally_reserved":0,"allocated":349741056,"data_stored":643855204,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":313260486},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,4,3,6,11,7],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 17:40:46 2023","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.39000000000000001,"15min":0.42599999999999999},"min":{"1min":0.154,"5min":0.154,"15min":0.154},"max":{"1min":5.593,"5min":5.593,"15min":5.593},"last":0.17699999999999999},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.41299999999999998,"15min":0.45700000000000002},"min":{"1min":0.193,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":3.1509999999999998,"5min":3.1509999999999998,"15min":3.1509999999999998},"last":0.21199999999999999}]},{"osd":2,"last update":"Tue Oct 3 17:40:46 2023","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.42799999999999999,"15min":0.53200000000000003},"min":{"1min":0.20699999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":1.4419999999999999,"5min":2.823,"15min":2.823},"last":0.26000000000000001},{"interface":"front","average":{"1min":0.378,"5min":0.434,"15min":0.51700000000000002},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":1.421,"5min":2.9239999999999999,"15min":2.9239999999999999},"last":0.193}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":245760,"data_stored":2339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":349450240,"data_stored":644763648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":342487040,"data_stored":633282579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":691834880,"data_stored":1277480979,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-03T17:41:33.120 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-03T17:34:47.281095+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=34, tm_sec=47, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=40, tm_sec=6, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T17:41:33.120 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-03T17:34:47.281095+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=34, tm_sec=47, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=40, tm_sec=6, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T17:41:33.120 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-03T17:34:47.281095+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=34, tm_sec=47, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=40, tm_sec=6, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T17:41:33.120 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-03T17:34:47.281095+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=34, tm_sec=47, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=40, tm_sec=6, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T17:41:33.120 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T17:41:51.799 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:41:51.798+0000 7f09e4851640 -1 reset not still connected to 0x557d75f2b2b0 2023-10-03T17:41:51.800 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:41:51.798+0000 7f09e4851640 -1 reset not still connected to 0x557d76253d40 2023-10-03T17:41:51.800 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:41:51.798+0000 7f09e4851640 -1 reset not still connected to 0x557d76621040 2023-10-03T17:41:51.800 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:41:51.798+0000 7f09e4851640 -1 reset not still connected to 0x557d7664e750 2023-10-03T17:41:51.800 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:41:51.798+0000 7f09e4851640 -1 reset not still connected to 0x557d7664ea90 2023-10-03T17:41:51.800 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:41:51.798+0000 7f09e4851640 -1 reset not still connected to 0x557d7664f380 2023-10-03T17:41:53.121 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T17:41:53.673 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-03T17:41:53.673 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-10-03T17:41:53.709 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":264,"stamp":"2023-10-03T17:41:52.050454+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20909251414,"num_objects":10235,"num_object_clones":5029,"num_object_copies":20470,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10235,"num_whiteouts":0,"num_read":182271,"num_read_kb":638743,"num_write":178064,"num_write_kb":712747,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":45430,"ondisk_log_size":45430,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2686184,"kb_used_data":1420172,"kb_used_omap":4,"kb_used_meta":1265979,"kb_avail":280429336,"statfs":{"total":289910292480,"available":287159640064,"internally_reserved":0,"allocated":1454256128,"data_stored":2723846258,"data_compressed":7460,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4782,"internal_metadata":1296362834},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,4,2,5,12,20,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":25825280,"num_objects":89,"num_object_clones":89,"num_object_copies":178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89,"num_whiteouts":0,"num_read":4542,"num_read_kb":17644,"num_write":4375,"num_write_kb":17500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002656"},"pg_stats":[{"pgid":"2.7","version":"48'936","reported_seq":1583,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-10-03T17:41:46.177814+0000","last_change":"2023-10-03T17:40:17.407011+0000","last_active":"2023-10-03T17:41:46.177814+0000","last_peered":"2023-10-03T17:41:46.177814+0000","last_clean":"2023-10-03T17:41:46.177814+0000","last_became_active":"2023-10-03T17:34:42.276624+0000","last_became_peered":"2023-10-03T17:34:42.276624+0000","last_unstale":"2023-10-03T17:41:46.177814+0000","last_undegraded":"2023-10-03T17:41:46.177814+0000","last_fullsized":"2023-10-03T17:41:46.177814+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'673","last_scrub_stamp":"2023-10-03T17:40:17.406813+0000","last_deep_scrub":"46'673","last_deep_scrub_stamp":"2023-10-03T17:40:17.406813+0000","last_clean_scrub_stamp":"2023-10-03T17:40:17.406813+0000","objects_scrubbed":7,"log_size":936,"log_dups_size":0,"ondisk_log_size":936,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T21:38:35.056889+0000","scrub_duration":0.013778076,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":385,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":1185,"num_read_kb":990,"num_write":30,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":903,"num_omap_keys":30,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'1577","reported_seq":2973,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-10-03T17:41:46.178567+0000","last_change":"2023-10-03T17:40:13.376519+0000","last_active":"2023-10-03T17:41:46.178567+0000","last_peered":"2023-10-03T17:41:46.178567+0000","last_clean":"2023-10-03T17:41:46.178567+0000","last_became_active":"2023-10-03T17:34:42.271201+0000","last_became_peered":"2023-10-03T17:34:42.271201+0000","last_unstale":"2023-10-03T17:41:46.178567+0000","last_undegraded":"2023-10-03T17:41:46.178567+0000","last_fullsized":"2023-10-03T17:41:46.178567+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'1129","last_scrub_stamp":"2023-10-03T17:40:13.376332+0000","last_deep_scrub":"46'1129","last_deep_scrub_stamp":"2023-10-03T17:40:13.376332+0000","last_clean_scrub_stamp":"2023-10-03T17:40:13.376332+0000","objects_scrubbed":11,"log_size":1577,"log_dups_size":0,"ondisk_log_size":1577,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T18:59:17.088585+0000","scrub_duration":0.015998675,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":2550,"num_read_kb":2241,"num_write":86,"num_write_kb":86,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2912,"num_omap_keys":93,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"48'15","reported_seq":92,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-10-03T17:41:01.210661+0000","last_change":"2023-10-03T17:40:25.496010+0000","last_active":"2023-10-03T17:41:01.210661+0000","last_peered":"2023-10-03T17:41:01.210661+0000","last_clean":"2023-10-03T17:41:01.210661+0000","last_became_active":"2023-10-03T17:34:42.276941+0000","last_became_peered":"2023-10-03T17:34:42.276941+0000","last_unstale":"2023-10-03T17:41:01.210661+0000","last_undegraded":"2023-10-03T17:41:01.210661+0000","last_fullsized":"2023-10-03T17:41:01.210661+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'14","last_scrub_stamp":"2023-10-03T17:40:25.495827+0000","last_deep_scrub":"46'14","last_deep_scrub_stamp":"2023-10-03T17:40:25.495827+0000","last_clean_scrub_stamp":"2023-10-03T17:40:25.495827+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:05:44.939663+0000","scrub_duration":0.0133078,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":300,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'1265","reported_seq":2315,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-10-03T17:41:46.178398+0000","last_change":"2023-10-03T17:40:26.513902+0000","last_active":"2023-10-03T17:41:46.178398+0000","last_peered":"2023-10-03T17:41:46.178398+0000","last_clean":"2023-10-03T17:41:46.178398+0000","last_became_active":"2023-10-03T17:34:42.276459+0000","last_became_peered":"2023-10-03T17:34:42.276459+0000","last_unstale":"2023-10-03T17:41:46.178398+0000","last_undegraded":"2023-10-03T17:41:46.178398+0000","last_fullsized":"2023-10-03T17:41:46.178398+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'964","last_scrub_stamp":"2023-10-03T17:40:26.513729+0000","last_deep_scrub":"46'964","last_deep_scrub_stamp":"2023-10-03T17:40:26.513729+0000","last_clean_scrub_stamp":"2023-10-03T17:40:26.513729+0000","objects_scrubbed":10,"log_size":1265,"log_dups_size":0,"ondisk_log_size":1265,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:41:33.182751+0000","scrub_duration":0.018190280999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1820,"num_read_kb":1534,"num_write":54,"num_write_kb":54,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1731,"num_omap_keys":80,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"48'47204","reported_seq":90657,"reported_epoch":48,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T17:41:48.622487+0000","last_change":"2023-10-03T17:40:11.716583+0000","last_active":"2023-10-03T17:41:48.622487+0000","last_peered":"2023-10-03T17:41:48.622487+0000","last_clean":"2023-10-03T17:41:48.622487+0000","last_became_active":"2023-10-03T17:34:48.310701+0000","last_became_peered":"2023-10-03T17:34:48.310701+0000","last_unstale":"2023-10-03T17:41:48.622487+0000","last_undegraded":"2023-10-03T17:41:48.622487+0000","last_fullsized":"2023-10-03T17:41:48.622487+0000","mapping_epoch":16,"log_start":"46'37119","ondisk_log_start":"46'37119","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_clean_scrub_stamp":"2023-10-03T17:34:47.281095+0000","objects_scrubbed":897,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 96s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040768800000000002,"stat_sum":{"num_bytes":5383995392,"num_objects":2616,"num_object_clones":1296,"num_object_copies":5232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2616,"num_whiteouts":0,"num_read":44576,"num_read_kb":162208,"num_write":45719,"num_write_kb":182876,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"48'1180","reported_seq":2035,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-10-03T17:41:48.307821+0000","last_change":"2023-10-03T17:40:08.675119+0000","last_active":"2023-10-03T17:41:48.307821+0000","last_peered":"2023-10-03T17:41:48.307821+0000","last_clean":"2023-10-03T17:41:48.307821+0000","last_became_active":"2023-10-03T17:34:42.270625+0000","last_became_peered":"2023-10-03T17:34:42.270625+0000","last_unstale":"2023-10-03T17:41:48.307821+0000","last_undegraded":"2023-10-03T17:41:48.307821+0000","last_fullsized":"2023-10-03T17:41:48.307821+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'841","last_scrub_stamp":"2023-10-03T17:40:08.674910+0000","last_deep_scrub":"46'841","last_deep_scrub_stamp":"2023-10-03T17:40:08.674910+0000","last_clean_scrub_stamp":"2023-10-03T17:40:08.674910+0000","objects_scrubbed":11,"log_size":1180,"log_dups_size":0,"ondisk_log_size":1180,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T21:05:42.790614+0000","scrub_duration":0.039189555000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":352,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":1593,"num_read_kb":1335,"num_write":57,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2564,"num_omap_keys":60,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"48'44078","reported_seq":85182,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-10-03T17:41:48.550217+0000","last_change":"2023-10-03T17:41:45.731649+0000","last_active":"2023-10-03T17:41:48.550217+0000","last_peered":"2023-10-03T17:41:48.550217+0000","last_clean":"2023-10-03T17:41:48.550217+0000","last_became_active":"2023-10-03T17:34:48.310555+0000","last_became_peered":"2023-10-03T17:34:48.310555+0000","last_unstale":"2023-10-03T17:41:48.550217+0000","last_undegraded":"2023-10-03T17:41:48.550217+0000","last_fullsized":"2023-10-03T17:41:48.550217+0000","mapping_epoch":16,"log_start":"46'34014","ondisk_log_start":"46'34014","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_clean_scrub_stamp":"2023-10-03T17:34:47.281095+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086289699999999995,"stat_sum":{"num_bytes":5076426752,"num_objects":2467,"num_object_clones":1225,"num_object_copies":4934,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2467,"num_whiteouts":0,"num_read":42232,"num_read_kb":153140,"num_write":42853,"num_write_kb":171412,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"48'50","reported_seq":683,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-10-03T17:41:46.172685+0000","last_change":"2023-10-03T17:40:17.641493+0000","last_active":"2023-10-03T17:41:46.172685+0000","last_peered":"2023-10-03T17:41:46.172685+0000","last_clean":"2023-10-03T17:41:46.172685+0000","last_became_active":"2023-10-03T17:34:42.278196+0000","last_became_peered":"2023-10-03T17:34:42.278196+0000","last_unstale":"2023-10-03T17:41:46.172685+0000","last_undegraded":"2023-10-03T17:41:46.172685+0000","last_fullsized":"2023-10-03T17:41:46.172685+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'30","last_scrub_stamp":"2023-10-03T17:40:17.640969+0000","last_deep_scrub":"46'30","last_deep_scrub_stamp":"2023-10-03T17:40:17.640969+0000","last_clean_scrub_stamp":"2023-10-03T17:40:17.640969+0000","objects_scrubbed":6,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:04:02.954896+0000","scrub_duration":0.071054565,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":218,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1171,"num_read_kb":985,"num_write":26,"num_write_kb":26,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":903,"num_omap_keys":30,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"48'45467","reported_seq":87530,"reported_epoch":48,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T17:41:48.601617+0000","last_change":"2023-10-03T17:40:10.712199+0000","last_active":"2023-10-03T17:41:48.601617+0000","last_peered":"2023-10-03T17:41:48.601617+0000","last_clean":"2023-10-03T17:41:48.601617+0000","last_became_active":"2023-10-03T17:34:48.310912+0000","last_became_peered":"2023-10-03T17:34:48.310912+0000","last_unstale":"2023-10-03T17:41:48.601617+0000","last_undegraded":"2023-10-03T17:41:48.601617+0000","last_fullsized":"2023-10-03T17:41:48.601617+0000","mapping_epoch":16,"log_start":"46'35411","ondisk_log_start":"46'35411","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_clean_scrub_stamp":"2023-10-03T17:34:47.281095+0000","objects_scrubbed":955,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 97s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045661900000000001,"stat_sum":{"num_bytes":5191819264,"num_objects":2510,"num_object_clones":1239,"num_object_copies":5020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2510,"num_whiteouts":0,"num_read":43119,"num_read_kb":157744,"num_write":44040,"num_write_kb":176160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"48'30","reported_seq":298,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-10-03T17:41:41.049245+0000","last_change":"2023-10-03T17:40:10.708344+0000","last_active":"2023-10-03T17:41:41.049245+0000","last_peered":"2023-10-03T17:41:41.049245+0000","last_clean":"2023-10-03T17:41:41.049245+0000","last_became_active":"2023-10-03T17:34:42.272306+0000","last_became_peered":"2023-10-03T17:34:42.272306+0000","last_unstale":"2023-10-03T17:41:41.049245+0000","last_undegraded":"2023-10-03T17:41:41.049245+0000","last_fullsized":"2023-10-03T17:41:41.049245+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'20","last_scrub_stamp":"2023-10-03T17:40:10.708060+0000","last_deep_scrub":"46'20","last_deep_scrub_stamp":"2023-10-03T17:40:10.708060+0000","last_clean_scrub_stamp":"2023-10-03T17:40:10.708060+0000","objects_scrubbed":7,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:03:09.097252+0000","scrub_duration":0.042279491000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":401,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":425,"num_read_kb":339,"num_write":17,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":301,"num_omap_keys":10,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"48'46488","reported_seq":88807,"reported_epoch":48,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T17:41:48.563557+0000","last_change":"2023-10-03T17:40:29.562114+0000","last_active":"2023-10-03T17:41:48.563557+0000","last_peered":"2023-10-03T17:41:48.563557+0000","last_clean":"2023-10-03T17:41:48.563557+0000","last_became_active":"2023-10-03T17:34:48.310612+0000","last_became_peered":"2023-10-03T17:34:48.310612+0000","last_unstale":"2023-10-03T17:41:48.563557+0000","last_undegraded":"2023-10-03T17:41:48.563557+0000","last_fullsized":"2023-10-03T17:41:48.563557+0000","mapping_epoch":16,"log_start":"46'36424","ondisk_log_start":"46'36424","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_clean_scrub_stamp":"2023-10-03T17:34:47.281095+0000","objects_scrubbed":732,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 79s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022375120000000001,"stat_sum":{"num_bytes":5256417299,"num_objects":2557,"num_object_clones":1269,"num_object_copies":5114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2557,"num_whiteouts":0,"num_read":43429,"num_read_kb":158116,"num_write":45032,"num_write_kb":180122,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"47'9","reported_seq":79,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-10-03T17:41:00.130118+0000","last_change":"2023-10-03T17:40:24.472734+0000","last_active":"2023-10-03T17:41:00.130118+0000","last_peered":"2023-10-03T17:41:00.130118+0000","last_clean":"2023-10-03T17:41:00.130118+0000","last_became_active":"2023-10-03T17:34:42.270194+0000","last_became_peered":"2023-10-03T17:34:42.270194+0000","last_unstale":"2023-10-03T17:41:00.130118+0000","last_undegraded":"2023-10-03T17:41:00.130118+0000","last_fullsized":"2023-10-03T17:41:00.130118+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'8","last_scrub_stamp":"2023-10-03T17:40:24.472573+0000","last_deep_scrub":"46'8","last_deep_scrub_stamp":"2023-10-03T17:40:24.472573+0000","last_clean_scrub_stamp":"2023-10-03T17:40:24.472573+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T18:28:04.735033+0000","scrub_duration":0.024990637999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":182,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":8,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"46'99","reported_seq":188,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-10-03T17:41:00.112863+0000","last_change":"2023-10-03T17:40:10.452341+0000","last_active":"2023-10-03T17:41:00.112863+0000","last_peered":"2023-10-03T17:41:00.112863+0000","last_clean":"2023-10-03T17:41:00.112863+0000","last_became_active":"2023-10-03T17:34:39.502393+0000","last_became_peered":"2023-10-03T17:34:39.502393+0000","last_unstale":"2023-10-03T17:41:00.112863+0000","last_undegraded":"2023-10-03T17:41:00.112863+0000","last_fullsized":"2023-10-03T17:41:00.112863+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'99","last_scrub_stamp":"2023-10-03T17:40:10.449099+0000","last_deep_scrub":"46'99","last_deep_scrub_stamp":"2023-10-03T17:40:10.449099+0000","last_clean_scrub_stamp":"2023-10-03T17:40:10.449099+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T20:18:04.958666+0000","scrub_duration":0.043225012,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":20908658707,"num_objects":10150,"num_object_clones":5029,"num_object_copies":20300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10150,"num_whiteouts":0,"num_read":173356,"num_read_kb":631208,"num_write":177644,"num_write_kb":710570,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1455067136,"data_stored":2724794406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40269,"ondisk_log_size":40269,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2339,"num_objects":83,"num_object_clones":0,"num_object_copies":166,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83,"num_whiteouts":0,"num_read":8809,"num_read_kb":7444,"num_write":283,"num_write_kb":283,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":4678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5062,"ondisk_log_size":5062,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":7460,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738461,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1331628,"kb_used_data":709348,"kb_used_omap":1,"kb_used_meta":622270,"kb_avail":93040212,"statfs":{"total":96636764160,"available":95273177088,"internally_reserved":0,"allocated":726372352,"data_stored":1360916086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":637204934},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,1,1,2,5,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 17:40:47 2023","interfaces":[{"interface":"back","average":{"1min":0.27900000000000003,"5min":0.40300000000000002,"15min":0.47799999999999998},"min":{"1min":0.17000000000000001,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":0.48799999999999999,"5min":4.4640000000000004,"15min":4.4640000000000004},"last":0.18099999999999999},{"interface":"front","average":{"1min":0.378,"5min":0.38100000000000001,"15min":0.498},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":1.9850000000000001,"5min":1.9850000000000001,"15min":1.9850000000000001},"last":0.36799999999999999}]},{"osd":1,"last update":"Tue Oct 3 17:40:47 2023","interfaces":[{"interface":"back","average":{"1min":0.28999999999999998,"5min":0.41399999999999998,"15min":0.50800000000000001},"min":{"1min":0.189,"5min":0.16600000000000001,"15min":0.16600000000000001},"max":{"1min":0.41499999999999998,"5min":4.4260000000000002,"15min":4.4260000000000002},"last":0.22600000000000001},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.378,"15min":0.52700000000000002},"min":{"1min":0.20799999999999999,"5min":0.192,"15min":0.192},"max":{"1min":2.0579999999999998,"5min":2.0579999999999998,"15min":2.0579999999999998},"last":0.246}]}]},{"osd":1,"up_from":8,"seq":34359738459,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":671396,"kb_used_data":351576,"kb_used_omap":1,"kb_used_meta":319806,"kb_avail":93700444,"statfs":{"total":96636764160,"available":95949254656,"internally_reserved":0,"allocated":360013824,"data_stored":675362456,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":327481798},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,1,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 17:40:59 2023","interfaces":[{"interface":"back","average":{"1min":0.32400000000000001,"5min":0.35299999999999998,"15min":0.46500000000000002},"min":{"1min":0.17100000000000001,"5min":0.161,"15min":0.161},"max":{"1min":1.8120000000000001,"5min":4.1879999999999997,"15min":4.1879999999999997},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.30599999999999999,"5min":0.34000000000000002,"15min":0.47199999999999998},"min":{"1min":0.17399999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.94099999999999995,"5min":1.532,"15min":1.532},"last":0.23200000000000001}]},{"osd":2,"last update":"Tue Oct 3 17:40:59 2023","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.373,"15min":0.48599999999999999},"min":{"1min":0.20499999999999999,"5min":0.159,"15min":0.159},"max":{"1min":3.0390000000000001,"5min":3.0390000000000001,"15min":3.0390000000000001},"last":0.315},{"interface":"front","average":{"1min":0.46800000000000003,"5min":0.375,"15min":0.51200000000000001},"min":{"1min":0.23400000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":3.0699999999999998,"5min":3.0699999999999998,"15min":3.0699999999999998},"last":0.216}]}]},{"osd":0,"up_from":8,"seq":34359738459,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":683160,"kb_used_data":359248,"kb_used_omap":1,"kb_used_meta":323902,"kb_avail":93688680,"statfs":{"total":96636764160,"available":95937208320,"internally_reserved":0,"allocated":367869952,"data_stored":687567716,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":331676102},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,2,4,15,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 17:41:48 2023","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":0.46200000000000002,"15min":0.44700000000000001},"min":{"1min":0.16500000000000001,"5min":0.154,"15min":0.154},"max":{"1min":5.7649999999999997,"5min":5.7649999999999997,"15min":5.7649999999999997},"last":0.217},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.40300000000000002,"15min":0.45000000000000001},"min":{"1min":0.19,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":1.6259999999999999,"5min":3.1509999999999998,"15min":3.1509999999999998},"last":0.19800000000000001}]},{"osd":2,"last update":"Tue Oct 3 17:41:48 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.39800000000000002,"15min":0.51400000000000001},"min":{"1min":0.20699999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.66000000000000003,"5min":2.823,"15min":2.823},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.34399999999999997,"5min":0.42599999999999999,"15min":0.503},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":0.57699999999999996,"5min":2.9239999999999999,"15min":2.9239999999999999},"last":0.379}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":245760,"data_stored":2339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":367656960,"data_stored":687222784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":359989248,"data_stored":675287059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":727420928,"data_stored":1362284563,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-03T17:41:53.711 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-03T17:34:47.281095+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=34, tm_sec=47, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=40, tm_sec=6, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T17:41:53.711 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-03T17:34:47.281095+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=34, tm_sec=47, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=40, tm_sec=6, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T17:41:53.711 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-03T17:34:47.281095+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=34, tm_sec=47, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=40, tm_sec=6, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T17:41:53.712 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-03T17:34:47.281095+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=34, tm_sec=47, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=40, tm_sec=6, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T17:41:53.712 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T17:42:13.712 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T17:42:14.278 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-03T17:42:14.279 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-10-03T17:42:14.313 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":278,"stamp":"2023-10-03T17:42:14.054497+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20977073993,"num_objects":13390,"num_object_clones":8164,"num_object_copies":26780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13390,"num_whiteouts":0,"num_read":195606,"num_read_kb":681202,"num_write":188379,"num_write_kb":753767,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":45791,"ondisk_log_size":45791,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2912188,"kb_used_data":1503328,"kb_used_omap":4,"kb_used_meta":1408827,"kb_avail":280203332,"statfs":{"total":289910292480,"available":286928211968,"internally_reserved":0,"allocated":1539407872,"data_stored":3704130493,"data_compressed":7460,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4782,"internal_metadata":1442639186},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,5,8,11,16,18,20,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":39941107,"num_objects":3116,"num_object_clones":3096,"num_object_copies":6232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3116,"num_whiteouts":0,"num_read":8922,"num_read_kb":25411,"num_write":5890,"num_write_kb":23320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.802833"},"pg_stats":[{"pgid":"2.7","version":"51'999","reported_seq":1779,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-10-03T17:42:13.452339+0000","last_change":"2023-10-03T17:40:17.407011+0000","last_active":"2023-10-03T17:42:13.452339+0000","last_peered":"2023-10-03T17:42:13.452339+0000","last_clean":"2023-10-03T17:42:13.452339+0000","last_became_active":"2023-10-03T17:34:42.276624+0000","last_became_peered":"2023-10-03T17:34:42.276624+0000","last_unstale":"2023-10-03T17:42:13.452339+0000","last_undegraded":"2023-10-03T17:42:13.452339+0000","last_fullsized":"2023-10-03T17:42:13.452339+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'673","last_scrub_stamp":"2023-10-03T17:40:17.406813+0000","last_deep_scrub":"46'673","last_deep_scrub_stamp":"2023-10-03T17:40:17.406813+0000","last_clean_scrub_stamp":"2023-10-03T17:40:17.406813+0000","objects_scrubbed":7,"log_size":999,"log_dups_size":0,"ondisk_log_size":999,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T21:38:35.056889+0000","scrub_duration":0.013778076,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1599,"num_read_kb":1329,"num_write":40,"num_write_kb":40,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":903,"num_omap_keys":30,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"51'1682","reported_seq":3418,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-10-03T17:42:13.458453+0000","last_change":"2023-10-03T17:40:13.376519+0000","last_active":"2023-10-03T17:42:13.458453+0000","last_peered":"2023-10-03T17:42:13.458453+0000","last_clean":"2023-10-03T17:42:13.458453+0000","last_became_active":"2023-10-03T17:34:42.271201+0000","last_became_peered":"2023-10-03T17:34:42.271201+0000","last_unstale":"2023-10-03T17:42:13.458453+0000","last_undegraded":"2023-10-03T17:42:13.458453+0000","last_fullsized":"2023-10-03T17:42:13.458453+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'1129","last_scrub_stamp":"2023-10-03T17:40:13.376332+0000","last_deep_scrub":"46'1129","last_deep_scrub_stamp":"2023-10-03T17:40:13.376332+0000","last_clean_scrub_stamp":"2023-10-03T17:40:13.376332+0000","objects_scrubbed":11,"log_size":1682,"log_dups_size":0,"ondisk_log_size":1682,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T18:59:17.088585+0000","scrub_duration":0.015998675,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":522,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":3404,"num_read_kb":2969,"num_write":104,"num_write_kb":104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2912,"num_omap_keys":93,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"50'21","reported_seq":101,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-10-03T17:42:01.409608+0000","last_change":"2023-10-03T17:40:25.496010+0000","last_active":"2023-10-03T17:42:01.409608+0000","last_peered":"2023-10-03T17:42:01.409608+0000","last_clean":"2023-10-03T17:42:01.409608+0000","last_became_active":"2023-10-03T17:34:42.276941+0000","last_became_peered":"2023-10-03T17:34:42.276941+0000","last_unstale":"2023-10-03T17:42:01.409608+0000","last_undegraded":"2023-10-03T17:42:01.409608+0000","last_fullsized":"2023-10-03T17:42:01.409608+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'14","last_scrub_stamp":"2023-10-03T17:40:25.495827+0000","last_deep_scrub":"46'14","last_deep_scrub_stamp":"2023-10-03T17:40:25.495827+0000","last_clean_scrub_stamp":"2023-10-03T17:40:25.495827+0000","objects_scrubbed":7,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:05:44.939663+0000","scrub_duration":0.0133078,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":604,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":14,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"51'1347","reported_seq":2634,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-10-03T17:42:13.455628+0000","last_change":"2023-10-03T17:40:26.513902+0000","last_active":"2023-10-03T17:42:13.455628+0000","last_peered":"2023-10-03T17:42:13.455628+0000","last_clean":"2023-10-03T17:42:13.455628+0000","last_became_active":"2023-10-03T17:34:42.276459+0000","last_became_peered":"2023-10-03T17:34:42.276459+0000","last_unstale":"2023-10-03T17:42:13.455628+0000","last_undegraded":"2023-10-03T17:42:13.455628+0000","last_fullsized":"2023-10-03T17:42:13.455628+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'964","last_scrub_stamp":"2023-10-03T17:40:26.513729+0000","last_deep_scrub":"46'964","last_deep_scrub_stamp":"2023-10-03T17:40:26.513729+0000","last_clean_scrub_stamp":"2023-10-03T17:40:26.513729+0000","objects_scrubbed":10,"log_size":1347,"log_dups_size":0,"ondisk_log_size":1347,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:41:33.182751+0000","scrub_duration":0.018190280999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":2434,"num_read_kb":2047,"num_write":66,"num_write_kb":66,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1731,"num_omap_keys":80,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"51'50765","reported_seq":96241,"reported_epoch":51,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T17:42:13.736231+0000","last_change":"2023-10-03T17:40:11.716583+0000","last_active":"2023-10-03T17:42:13.736231+0000","last_peered":"2023-10-03T17:42:13.736231+0000","last_clean":"2023-10-03T17:42:13.736231+0000","last_became_active":"2023-10-03T17:34:48.310701+0000","last_became_peered":"2023-10-03T17:34:48.310701+0000","last_unstale":"2023-10-03T17:42:13.736231+0000","last_undegraded":"2023-10-03T17:42:13.736231+0000","last_fullsized":"2023-10-03T17:42:13.736231+0000","mapping_epoch":16,"log_start":"48'40722","ondisk_log_start":"48'40722","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_clean_scrub_stamp":"2023-10-03T17:34:47.281095+0000","objects_scrubbed":1032,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 122s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040768800000000002,"stat_sum":{"num_bytes":5401714688,"num_objects":3428,"num_object_clones":2108,"num_object_copies":6856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3428,"num_whiteouts":0,"num_read":47382,"num_read_kb":173004,"num_write":48401,"num_write_kb":193604,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"51'1273","reported_seq":2305,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-10-03T17:42:12.179276+0000","last_change":"2023-10-03T17:40:08.675119+0000","last_active":"2023-10-03T17:42:12.179276+0000","last_peered":"2023-10-03T17:42:12.179276+0000","last_clean":"2023-10-03T17:42:12.179276+0000","last_became_active":"2023-10-03T17:34:42.270625+0000","last_became_peered":"2023-10-03T17:34:42.270625+0000","last_unstale":"2023-10-03T17:42:12.179276+0000","last_undegraded":"2023-10-03T17:42:12.179276+0000","last_fullsized":"2023-10-03T17:42:12.179276+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'841","last_scrub_stamp":"2023-10-03T17:40:08.674910+0000","last_deep_scrub":"46'841","last_deep_scrub_stamp":"2023-10-03T17:40:08.674910+0000","last_clean_scrub_stamp":"2023-10-03T17:40:08.674910+0000","objects_scrubbed":11,"log_size":1273,"log_dups_size":0,"ondisk_log_size":1273,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T21:05:42.790614+0000","scrub_duration":0.039189555000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":605,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":2147,"num_read_kb":1790,"num_write":74,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2564,"num_omap_keys":60,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"51'47344","reported_seq":90149,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-10-03T17:42:13.663923+0000","last_change":"2023-10-03T17:42:12.873942+0000","last_active":"2023-10-03T17:42:13.663923+0000","last_peered":"2023-10-03T17:42:13.663923+0000","last_clean":"2023-10-03T17:42:13.663923+0000","last_became_active":"2023-10-03T17:34:48.310555+0000","last_became_peered":"2023-10-03T17:34:48.310555+0000","last_unstale":"2023-10-03T17:42:13.663923+0000","last_undegraded":"2023-10-03T17:42:13.663923+0000","last_fullsized":"2023-10-03T17:42:13.663923+0000","mapping_epoch":16,"log_start":"46'37319","ondisk_log_start":"46'37319","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_clean_scrub_stamp":"2023-10-03T17:34:47.281095+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086289699999999995,"stat_sum":{"num_bytes":5092986880,"num_objects":3232,"num_object_clones":1990,"num_object_copies":6464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3232,"num_whiteouts":0,"num_read":44683,"num_read_kb":162592,"num_write":45354,"num_write_kb":181416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"50'66","reported_seq":833,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-10-03T17:42:11.768759+0000","last_change":"2023-10-03T17:40:17.641493+0000","last_active":"2023-10-03T17:42:11.768759+0000","last_peered":"2023-10-03T17:42:11.768759+0000","last_clean":"2023-10-03T17:42:11.768759+0000","last_became_active":"2023-10-03T17:34:42.278196+0000","last_became_peered":"2023-10-03T17:34:42.278196+0000","last_unstale":"2023-10-03T17:42:11.768759+0000","last_undegraded":"2023-10-03T17:42:11.768759+0000","last_fullsized":"2023-10-03T17:42:11.768759+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'30","last_scrub_stamp":"2023-10-03T17:40:17.640969+0000","last_deep_scrub":"46'30","last_deep_scrub_stamp":"2023-10-03T17:40:17.640969+0000","last_clean_scrub_stamp":"2023-10-03T17:40:17.640969+0000","objects_scrubbed":6,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:04:02.954896+0000","scrub_duration":0.071054565,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":318,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":1585,"num_read_kb":1324,"num_write":37,"num_write_kb":37,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":903,"num_omap_keys":30,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"51'48916","reported_seq":92750,"reported_epoch":51,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T17:42:13.701386+0000","last_change":"2023-10-03T17:40:10.712199+0000","last_active":"2023-10-03T17:42:13.701386+0000","last_peered":"2023-10-03T17:42:13.701386+0000","last_clean":"2023-10-03T17:42:13.701386+0000","last_became_active":"2023-10-03T17:34:48.310912+0000","last_became_peered":"2023-10-03T17:34:48.310912+0000","last_unstale":"2023-10-03T17:42:13.701386+0000","last_undegraded":"2023-10-03T17:42:13.701386+0000","last_fullsized":"2023-10-03T17:42:13.701386+0000","mapping_epoch":16,"log_start":"46'38817","ondisk_log_start":"46'38817","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_clean_scrub_stamp":"2023-10-03T17:34:47.281095+0000","objects_scrubbed":1300,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 122s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045661900000000001,"stat_sum":{"num_bytes":5208903680,"num_objects":3281,"num_object_clones":2010,"num_object_copies":6562,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3281,"num_whiteouts":0,"num_read":45635,"num_read_kb":167480,"num_write":46550,"num_write_kb":186200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"49'35","reported_seq":349,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-10-03T17:42:06.663739+0000","last_change":"2023-10-03T17:40:10.708344+0000","last_active":"2023-10-03T17:42:06.663739+0000","last_peered":"2023-10-03T17:42:06.663739+0000","last_clean":"2023-10-03T17:42:06.663739+0000","last_became_active":"2023-10-03T17:34:42.272306+0000","last_became_peered":"2023-10-03T17:34:42.272306+0000","last_unstale":"2023-10-03T17:42:06.663739+0000","last_undegraded":"2023-10-03T17:42:06.663739+0000","last_fullsized":"2023-10-03T17:42:06.663739+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'20","last_scrub_stamp":"2023-10-03T17:40:10.708060+0000","last_deep_scrub":"46'20","last_deep_scrub_stamp":"2023-10-03T17:40:10.708060+0000","last_clean_scrub_stamp":"2023-10-03T17:40:10.708060+0000","objects_scrubbed":7,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:03:09.097252+0000","scrub_duration":0.042279491000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":451,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":563,"num_read_kb":452,"num_write":21,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":301,"num_omap_keys":10,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"51'49925","reported_seq":94052,"reported_epoch":51,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T17:42:13.630744+0000","last_change":"2023-10-03T17:40:29.562114+0000","last_active":"2023-10-03T17:42:13.630744+0000","last_peered":"2023-10-03T17:42:13.630744+0000","last_clean":"2023-10-03T17:42:13.630744+0000","last_became_active":"2023-10-03T17:34:48.310612+0000","last_became_peered":"2023-10-03T17:34:48.310612+0000","last_unstale":"2023-10-03T17:42:13.630744+0000","last_undegraded":"2023-10-03T17:42:13.630744+0000","last_fullsized":"2023-10-03T17:42:13.630744+0000","mapping_epoch":16,"log_start":"46'39834","ondisk_log_start":"46'39834","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_clean_scrub_stamp":"2023-10-03T17:34:47.281095+0000","objects_scrubbed":934,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 104s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022375120000000001,"stat_sum":{"num_bytes":5272875027,"num_objects":3344,"num_object_clones":2056,"num_object_copies":6688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3344,"num_whiteouts":0,"num_read":46003,"num_read_kb":168104,"num_write":47574,"num_write_kb":190290,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"50'11","reported_seq":84,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-10-03T17:42:01.309426+0000","last_change":"2023-10-03T17:40:24.472734+0000","last_active":"2023-10-03T17:42:01.309426+0000","last_peered":"2023-10-03T17:42:01.309426+0000","last_clean":"2023-10-03T17:42:01.309426+0000","last_became_active":"2023-10-03T17:34:42.270194+0000","last_became_peered":"2023-10-03T17:34:42.270194+0000","last_unstale":"2023-10-03T17:42:01.309426+0000","last_undegraded":"2023-10-03T17:42:01.309426+0000","last_fullsized":"2023-10-03T17:42:01.309426+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'8","last_scrub_stamp":"2023-10-03T17:40:24.472573+0000","last_deep_scrub":"46'8","last_deep_scrub_stamp":"2023-10-03T17:40:24.472573+0000","last_clean_scrub_stamp":"2023-10-03T17:40:24.472573+0000","objects_scrubbed":4,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T18:28:04.735033+0000","scrub_duration":0.024990637999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":283,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":8,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"46'99","reported_seq":191,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-10-03T17:42:01.252364+0000","last_change":"2023-10-03T17:40:10.452341+0000","last_active":"2023-10-03T17:42:01.252364+0000","last_peered":"2023-10-03T17:42:01.252364+0000","last_clean":"2023-10-03T17:42:01.252364+0000","last_became_active":"2023-10-03T17:34:39.502393+0000","last_became_peered":"2023-10-03T17:34:39.502393+0000","last_unstale":"2023-10-03T17:42:01.252364+0000","last_undegraded":"2023-10-03T17:42:01.252364+0000","last_fullsized":"2023-10-03T17:42:01.252364+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'99","last_scrub_stamp":"2023-10-03T17:40:10.449099+0000","last_deep_scrub":"46'99","last_deep_scrub_stamp":"2023-10-03T17:40:10.449099+0000","last_clean_scrub_stamp":"2023-10-03T17:40:10.449099+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T20:18:04.958666+0000","scrub_duration":0.043225012,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":20976480275,"num_objects":13285,"num_object_clones":8164,"num_object_copies":26570,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13285,"num_whiteouts":0,"num_read":183703,"num_read_kb":671180,"num_write":187879,"num_write_kb":751510,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1538949120,"data_stored":3727732774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40258,"ondisk_log_size":40258,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":3350,"num_objects":103,"num_object_clones":0,"num_object_copies":206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":103,"num_whiteouts":0,"num_read":11797,"num_read_kb":9931,"num_write":363,"num_write_kb":363,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":655360,"data_stored":6700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5434,"ondisk_log_size":5434,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":7460,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738466,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1449776,"kb_used_data":751208,"kb_used_omap":1,"kb_used_meta":698558,"kb_avail":92922064,"statfs":{"total":96636764160,"available":95152193536,"internally_reserved":0,"allocated":769236992,"data_stored":1851895272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":715323846},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,4,4,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 17:41:48 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.39400000000000002,"15min":0.46800000000000003},"min":{"1min":0.158,"5min":0.158,"15min":0.14299999999999999},"max":{"1min":1.8500000000000001,"5min":4.4640000000000004,"15min":4.4640000000000004},"last":0.253},{"interface":"front","average":{"1min":0.41999999999999998,"5min":0.38,"15min":0.48899999999999999},"min":{"1min":0.19600000000000001,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":1.901,"5min":1.9850000000000001,"15min":1.9850000000000001},"last":0.497}]},{"osd":1,"last update":"Tue Oct 3 17:41:48 2023","interfaces":[{"interface":"back","average":{"1min":0.29699999999999999,"5min":0.39000000000000001,"15min":0.49099999999999999},"min":{"1min":0.17599999999999999,"5min":0.17599999999999999,"15min":0.16600000000000001},"max":{"1min":0.69999999999999996,"5min":4.4260000000000002,"15min":4.4260000000000002},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.318,"5min":0.35399999999999998,"15min":0.50800000000000001},"min":{"1min":0.223,"5min":0.192,"15min":0.192},"max":{"1min":0.48299999999999998,"5min":2.0579999999999998,"15min":2.0579999999999998},"last":0.748}]}]},{"osd":1,"up_from":8,"seq":34359738464,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731932,"kb_used_data":371924,"kb_used_omap":1,"kb_used_meta":359998,"kb_avail":93639908,"statfs":{"total":96636764160,"available":95887265792,"internally_reserved":0,"allocated":380850176,"data_stored":915228930,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":368638406},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,4,4,3,7,7,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 17:42:02 2023","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":0.36699999999999999,"15min":0.45600000000000002},"min":{"1min":0.17499999999999999,"5min":0.161,"15min":0.161},"max":{"1min":1.2809999999999999,"5min":4.1879999999999997,"15min":4.1879999999999997},"last":0.28699999999999998},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.33700000000000002,"15min":0.46300000000000002},"min":{"1min":0.17599999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":2.8399999999999999,"5min":2.8399999999999999,"15min":2.8399999999999999},"last":0.44700000000000001}]},{"osd":2,"last update":"Tue Oct 3 17:42:02 2023","interfaces":[{"interface":"back","average":{"1min":0.33300000000000002,"5min":0.35999999999999999,"15min":0.47199999999999998},"min":{"1min":0.20100000000000001,"5min":0.17799999999999999,"15min":0.159},"max":{"1min":0.65100000000000002,"5min":3.0390000000000001,"15min":3.0390000000000001},"last":0.32300000000000001},{"interface":"front","average":{"1min":0.29999999999999999,"5min":0.36199999999999999,"15min":0.49299999999999999},"min":{"1min":0.216,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":0.51300000000000001,"5min":3.0699999999999998,"15min":3.0699999999999998},"last":0.255}]}]},{"osd":0,"up_from":8,"seq":34359738464,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":730480,"kb_used_data":380196,"kb_used_omap":1,"kb_used_meta":350270,"kb_avail":93641360,"statfs":{"total":96636764160,"available":95888752640,"internally_reserved":0,"allocated":389320704,"data_stored":937006291,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":358676934},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,3,4,5,7,14,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 17:41:48 2023","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":0.46200000000000002,"15min":0.44700000000000001},"min":{"1min":0.16500000000000001,"5min":0.154,"15min":0.154},"max":{"1min":5.7649999999999997,"5min":5.7649999999999997,"15min":5.7649999999999997},"last":0.316},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.40300000000000002,"15min":0.45000000000000001},"min":{"1min":0.19,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":1.6259999999999999,"5min":3.1509999999999998,"15min":3.1509999999999998},"last":0.71299999999999997}]},{"osd":2,"last update":"Tue Oct 3 17:41:48 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.39800000000000002,"15min":0.51400000000000001},"min":{"1min":0.20699999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.66000000000000003,"5min":2.823,"15min":2.823},"last":1.5009999999999999},{"interface":"front","average":{"1min":0.34399999999999997,"5min":0.42599999999999999,"15min":0.503},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":0.57699999999999996,"5min":2.9239999999999999,"15min":2.9239999999999999},"last":1.5509999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":225280,"data_stored":2298,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":3350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":1052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":388919296,"data_stored":941957120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":380633088,"data_stored":923136019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":769396736,"data_stored":1862639635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-03T17:42:14.315 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-03T17:34:47.281095+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=34, tm_sec=47, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=40, tm_sec=6, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T17:42:14.315 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-03T17:34:47.281095+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=34, tm_sec=47, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=40, tm_sec=6, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T17:42:14.316 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-03T17:34:47.281095+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=34, tm_sec=47, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=40, tm_sec=6, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T17:42:14.316 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-03T17:34:47.281095+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=34, tm_sec=47, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=40, tm_sec=6, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T17:42:14.316 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T17:42:14.562 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:42:14.562+0000 7f09e4851640 -1 reset not still connected to 0x557d75f2b2b0 2023-10-03T17:42:14.562 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:42:14.562+0000 7f09e4851640 -1 reset not still connected to 0x557d76253d40 2023-10-03T17:42:14.563 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:42:14.562+0000 7f09e4851640 -1 reset not still connected to 0x557d76621040 2023-10-03T17:42:14.563 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:42:14.562+0000 7f09e4851640 -1 reset not still connected to 0x557d7664e750 2023-10-03T17:42:14.563 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:42:14.562+0000 7f09e4851640 -1 reset not still connected to 0x557d7664ea90 2023-10-03T17:42:14.563 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:42:14.562+0000 7f09e4851640 -1 reset not still connected to 0x557d7664f380 2023-10-03T17:42:14.563 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:42:14.562+0000 7f09e4851640 -1 reset not still connected to 0x557d7664f930 2023-10-03T17:42:14.563 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:42:14.562+0000 7f09e4851640 -1 reset not still connected to 0x557d76681ba0 2023-10-03T17:42:14.563 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:42:14.562+0000 7f09e4851640 -1 reset not still connected to 0x557d766a40d0 2023-10-03T17:42:14.563 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:42:14.562+0000 7f09e4851640 -1 reset not still connected to 0x557d766b56c0 2023-10-03T17:42:14.564 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:42:14.562+0000 7f09e4851640 -1 reset not still connected to 0x557d76eca270 2023-10-03T17:42:14.564 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:42:14.562+0000 7f09e4851640 -1 reset not still connected to 0x557d76f93ba0 2023-10-03T17:42:34.318 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T17:42:34.798 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-03T17:42:34.800 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-10-03T17:42:34.828 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":288,"stamp":"2023-10-03T17:42:34.057853+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":21034221385,"num_objects":14997,"num_object_clones":9771,"num_object_copies":29994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14997,"num_whiteouts":0,"num_read":203795,"num_read_kb":712854,"num_write":196349,"num_write_kb":785647,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":46043,"ondisk_log_size":46043,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3186484,"kb_used_data":1567196,"kb_used_omap":4,"kb_used_meta":1619259,"kb_avail":279929036,"statfs":{"total":289910292480,"available":286647332864,"internally_reserved":0,"allocated":1604808704,"data_stored":4252953533,"data_compressed":7460,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4782,"internal_metadata":1658121554},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,5,12,6,17,5,12,11],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":42188800,"num_objects":999,"num_object_clones":999,"num_object_copies":1998,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":999,"num_whiteouts":0,"num_read":6177,"num_read_kb":23848,"num_write":6052,"num_write_kb":24208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001983"},"pg_stats":[{"pgid":"2.7","version":"51'1054","reported_seq":1846,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-10-03T17:42:32.249585+0000","last_change":"2023-10-03T17:40:17.407011+0000","last_active":"2023-10-03T17:42:32.249585+0000","last_peered":"2023-10-03T17:42:32.249585+0000","last_clean":"2023-10-03T17:42:32.249585+0000","last_became_active":"2023-10-03T17:34:42.276624+0000","last_became_peered":"2023-10-03T17:34:42.276624+0000","last_unstale":"2023-10-03T17:42:32.249585+0000","last_undegraded":"2023-10-03T17:42:32.249585+0000","last_fullsized":"2023-10-03T17:42:32.249585+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'673","last_scrub_stamp":"2023-10-03T17:40:17.406813+0000","last_deep_scrub":"46'673","last_deep_scrub_stamp":"2023-10-03T17:40:17.406813+0000","last_clean_scrub_stamp":"2023-10-03T17:40:17.406813+0000","objects_scrubbed":7,"log_size":1054,"log_dups_size":0,"ondisk_log_size":1054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T21:38:35.056889+0000","scrub_duration":0.013778076,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1599,"num_read_kb":1329,"num_write":40,"num_write_kb":40,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":903,"num_omap_keys":30,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"51'1773","reported_seq":3529,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-10-03T17:42:32.249822+0000","last_change":"2023-10-03T17:40:13.376519+0000","last_active":"2023-10-03T17:42:32.249822+0000","last_peered":"2023-10-03T17:42:32.249822+0000","last_clean":"2023-10-03T17:42:32.249822+0000","last_became_active":"2023-10-03T17:34:42.271201+0000","last_became_peered":"2023-10-03T17:34:42.271201+0000","last_unstale":"2023-10-03T17:42:32.249822+0000","last_undegraded":"2023-10-03T17:42:32.249822+0000","last_fullsized":"2023-10-03T17:42:32.249822+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'1129","last_scrub_stamp":"2023-10-03T17:40:13.376332+0000","last_deep_scrub":"46'1129","last_deep_scrub_stamp":"2023-10-03T17:40:13.376332+0000","last_clean_scrub_stamp":"2023-10-03T17:40:13.376332+0000","objects_scrubbed":11,"log_size":1773,"log_dups_size":0,"ondisk_log_size":1773,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T18:59:17.088585+0000","scrub_duration":0.015998675,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":522,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":3404,"num_read_kb":2969,"num_write":104,"num_write_kb":104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2912,"num_omap_keys":93,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"50'21","reported_seq":101,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-10-03T17:42:01.409608+0000","last_change":"2023-10-03T17:40:25.496010+0000","last_active":"2023-10-03T17:42:01.409608+0000","last_peered":"2023-10-03T17:42:01.409608+0000","last_clean":"2023-10-03T17:42:01.409608+0000","last_became_active":"2023-10-03T17:34:42.276941+0000","last_became_peered":"2023-10-03T17:34:42.276941+0000","last_unstale":"2023-10-03T17:42:01.409608+0000","last_undegraded":"2023-10-03T17:42:01.409608+0000","last_fullsized":"2023-10-03T17:42:01.409608+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'14","last_scrub_stamp":"2023-10-03T17:40:25.495827+0000","last_deep_scrub":"46'14","last_deep_scrub_stamp":"2023-10-03T17:40:25.495827+0000","last_clean_scrub_stamp":"2023-10-03T17:40:25.495827+0000","objects_scrubbed":7,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:05:44.939663+0000","scrub_duration":0.0133078,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":604,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":14,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"51'1420","reported_seq":2723,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-10-03T17:42:32.249415+0000","last_change":"2023-10-03T17:40:26.513902+0000","last_active":"2023-10-03T17:42:32.249415+0000","last_peered":"2023-10-03T17:42:32.249415+0000","last_clean":"2023-10-03T17:42:32.249415+0000","last_became_active":"2023-10-03T17:34:42.276459+0000","last_became_peered":"2023-10-03T17:34:42.276459+0000","last_unstale":"2023-10-03T17:42:32.249415+0000","last_undegraded":"2023-10-03T17:42:32.249415+0000","last_fullsized":"2023-10-03T17:42:32.249415+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'964","last_scrub_stamp":"2023-10-03T17:40:26.513729+0000","last_deep_scrub":"46'964","last_deep_scrub_stamp":"2023-10-03T17:40:26.513729+0000","last_clean_scrub_stamp":"2023-10-03T17:40:26.513729+0000","objects_scrubbed":10,"log_size":1420,"log_dups_size":0,"ondisk_log_size":1420,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:41:33.182751+0000","scrub_duration":0.018190280999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":2434,"num_read_kb":2047,"num_write":66,"num_write_kb":66,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1731,"num_omap_keys":80,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"51'53365","reported_seq":100499,"reported_epoch":51,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T17:42:33.815615+0000","last_change":"2023-10-03T17:40:11.716583+0000","last_active":"2023-10-03T17:42:33.815615+0000","last_peered":"2023-10-03T17:42:33.815615+0000","last_clean":"2023-10-03T17:42:33.815615+0000","last_became_active":"2023-10-03T17:34:48.310701+0000","last_became_peered":"2023-10-03T17:34:48.310701+0000","last_unstale":"2023-10-03T17:42:33.815615+0000","last_undegraded":"2023-10-03T17:42:33.815615+0000","last_fullsized":"2023-10-03T17:42:33.815615+0000","mapping_epoch":16,"log_start":"48'43326","ondisk_log_start":"48'43326","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_clean_scrub_stamp":"2023-10-03T17:34:47.281095+0000","objects_scrubbed":1231,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 142s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040768800000000002,"stat_sum":{"num_bytes":5416796160,"num_objects":3850,"num_object_clones":2530,"num_object_copies":7700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3850,"num_whiteouts":0,"num_read":49447,"num_read_kb":180988,"num_write":50460,"num_write_kb":201840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"51'1373","reported_seq":2421,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-10-03T17:42:33.758823+0000","last_change":"2023-10-03T17:40:08.675119+0000","last_active":"2023-10-03T17:42:33.758823+0000","last_peered":"2023-10-03T17:42:33.758823+0000","last_clean":"2023-10-03T17:42:33.758823+0000","last_became_active":"2023-10-03T17:34:42.270625+0000","last_became_peered":"2023-10-03T17:34:42.270625+0000","last_unstale":"2023-10-03T17:42:33.758823+0000","last_undegraded":"2023-10-03T17:42:33.758823+0000","last_fullsized":"2023-10-03T17:42:33.758823+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'841","last_scrub_stamp":"2023-10-03T17:40:08.674910+0000","last_deep_scrub":"46'841","last_deep_scrub_stamp":"2023-10-03T17:40:08.674910+0000","last_clean_scrub_stamp":"2023-10-03T17:40:08.674910+0000","objects_scrubbed":11,"log_size":1373,"log_dups_size":0,"ondisk_log_size":1373,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T21:05:42.790614+0000","scrub_duration":0.039189555000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":605,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":2147,"num_read_kb":1790,"num_write":74,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2564,"num_omap_keys":60,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"51'49661","reported_seq":94101,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-10-03T17:42:33.747572+0000","last_change":"2023-10-03T17:42:32.008420+0000","last_active":"2023-10-03T17:42:33.747572+0000","last_peered":"2023-10-03T17:42:33.747572+0000","last_clean":"2023-10-03T17:42:33.747572+0000","last_became_active":"2023-10-03T17:34:48.310555+0000","last_became_peered":"2023-10-03T17:34:48.310555+0000","last_unstale":"2023-10-03T17:42:33.747572+0000","last_undegraded":"2023-10-03T17:42:33.747572+0000","last_fullsized":"2023-10-03T17:42:33.747572+0000","mapping_epoch":16,"log_start":"48'39621","ondisk_log_start":"48'39621","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_clean_scrub_stamp":"2023-10-03T17:34:47.281095+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086289699999999995,"stat_sum":{"num_bytes":5105184768,"num_objects":3605,"num_object_clones":2363,"num_object_copies":7210,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3605,"num_whiteouts":0,"num_read":46682,"num_read_kb":170340,"num_write":47298,"num_write_kb":189192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"50'66","reported_seq":845,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-10-03T17:42:32.279917+0000","last_change":"2023-10-03T17:40:17.641493+0000","last_active":"2023-10-03T17:42:32.279917+0000","last_peered":"2023-10-03T17:42:32.279917+0000","last_clean":"2023-10-03T17:42:32.279917+0000","last_became_active":"2023-10-03T17:34:42.278196+0000","last_became_peered":"2023-10-03T17:34:42.278196+0000","last_unstale":"2023-10-03T17:42:32.279917+0000","last_undegraded":"2023-10-03T17:42:32.279917+0000","last_fullsized":"2023-10-03T17:42:32.279917+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'30","last_scrub_stamp":"2023-10-03T17:40:17.640969+0000","last_deep_scrub":"46'30","last_deep_scrub_stamp":"2023-10-03T17:40:17.640969+0000","last_clean_scrub_stamp":"2023-10-03T17:40:17.640969+0000","objects_scrubbed":6,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:04:02.954896+0000","scrub_duration":0.071054565,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":318,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":1585,"num_read_kb":1324,"num_write":37,"num_write_kb":37,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":903,"num_omap_keys":30,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"51'51458","reported_seq":96919,"reported_epoch":51,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T17:42:33.811106+0000","last_change":"2023-10-03T17:40:10.712199+0000","last_active":"2023-10-03T17:42:33.811106+0000","last_peered":"2023-10-03T17:42:33.811106+0000","last_clean":"2023-10-03T17:42:33.811106+0000","last_became_active":"2023-10-03T17:34:48.310912+0000","last_became_peered":"2023-10-03T17:34:48.310912+0000","last_unstale":"2023-10-03T17:42:33.811106+0000","last_undegraded":"2023-10-03T17:42:33.811106+0000","last_fullsized":"2023-10-03T17:42:33.811106+0000","mapping_epoch":16,"log_start":"48'41425","ondisk_log_start":"48'41425","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_clean_scrub_stamp":"2023-10-03T17:34:47.281095+0000","objects_scrubbed":1631,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 143s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045661900000000001,"stat_sum":{"num_bytes":5223346176,"num_objects":3694,"num_object_clones":2423,"num_object_copies":7388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3694,"num_whiteouts":0,"num_read":47648,"num_read_kb":175244,"num_write":48485,"num_write_kb":193940,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"49'35","reported_seq":353,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-10-03T17:42:27.220365+0000","last_change":"2023-10-03T17:40:10.708344+0000","last_active":"2023-10-03T17:42:27.220365+0000","last_peered":"2023-10-03T17:42:27.220365+0000","last_clean":"2023-10-03T17:42:27.220365+0000","last_became_active":"2023-10-03T17:34:42.272306+0000","last_became_peered":"2023-10-03T17:34:42.272306+0000","last_unstale":"2023-10-03T17:42:27.220365+0000","last_undegraded":"2023-10-03T17:42:27.220365+0000","last_fullsized":"2023-10-03T17:42:27.220365+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'20","last_scrub_stamp":"2023-10-03T17:40:10.708060+0000","last_deep_scrub":"46'20","last_deep_scrub_stamp":"2023-10-03T17:40:10.708060+0000","last_clean_scrub_stamp":"2023-10-03T17:40:10.708060+0000","objects_scrubbed":7,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:03:09.097252+0000","scrub_duration":0.042279491000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":451,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":563,"num_read_kb":452,"num_write":21,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":301,"num_omap_keys":10,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"51'52515","reported_seq":98380,"reported_epoch":51,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T17:42:33.758484+0000","last_change":"2023-10-03T17:40:29.562114+0000","last_active":"2023-10-03T17:42:33.758484+0000","last_peered":"2023-10-03T17:42:33.758484+0000","last_clean":"2023-10-03T17:42:33.758484+0000","last_became_active":"2023-10-03T17:34:48.310612+0000","last_became_peered":"2023-10-03T17:34:48.310612+0000","last_unstale":"2023-10-03T17:42:33.758484+0000","last_undegraded":"2023-10-03T17:42:33.758484+0000","last_fullsized":"2023-10-03T17:42:33.758484+0000","mapping_epoch":16,"log_start":"48'42436","ondisk_log_start":"48'42436","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_clean_scrub_stamp":"2023-10-03T17:34:47.281095+0000","objects_scrubbed":1217,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 124s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022375120000000001,"stat_sum":{"num_bytes":5288300563,"num_objects":3743,"num_object_clones":2455,"num_object_copies":7486,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3743,"num_whiteouts":0,"num_read":48115,"num_read_kb":176260,"num_write":49606,"num_write_kb":198418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"50'11","reported_seq":84,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-10-03T17:42:01.309426+0000","last_change":"2023-10-03T17:40:24.472734+0000","last_active":"2023-10-03T17:42:01.309426+0000","last_peered":"2023-10-03T17:42:01.309426+0000","last_clean":"2023-10-03T17:42:01.309426+0000","last_became_active":"2023-10-03T17:34:42.270194+0000","last_became_peered":"2023-10-03T17:34:42.270194+0000","last_unstale":"2023-10-03T17:42:01.309426+0000","last_undegraded":"2023-10-03T17:42:01.309426+0000","last_fullsized":"2023-10-03T17:42:01.309426+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'8","last_scrub_stamp":"2023-10-03T17:40:24.472573+0000","last_deep_scrub":"46'8","last_deep_scrub_stamp":"2023-10-03T17:40:24.472573+0000","last_clean_scrub_stamp":"2023-10-03T17:40:24.472573+0000","objects_scrubbed":4,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T18:28:04.735033+0000","scrub_duration":0.024990637999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":283,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":8,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"46'99","reported_seq":191,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-10-03T17:42:01.252364+0000","last_change":"2023-10-03T17:40:10.452341+0000","last_active":"2023-10-03T17:42:01.252364+0000","last_peered":"2023-10-03T17:42:01.252364+0000","last_clean":"2023-10-03T17:42:01.252364+0000","last_became_active":"2023-10-03T17:34:39.502393+0000","last_became_peered":"2023-10-03T17:34:39.502393+0000","last_unstale":"2023-10-03T17:42:01.252364+0000","last_undegraded":"2023-10-03T17:42:01.252364+0000","last_fullsized":"2023-10-03T17:42:01.252364+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'99","last_scrub_stamp":"2023-10-03T17:40:10.449099+0000","last_deep_scrub":"46'99","last_deep_scrub_stamp":"2023-10-03T17:40:10.449099+0000","last_clean_scrub_stamp":"2023-10-03T17:40:10.449099+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T20:18:04.958666+0000","scrub_duration":0.043225012,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":21033627667,"num_objects":14892,"num_object_clones":9771,"num_object_copies":29784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14892,"num_whiteouts":0,"num_read":191892,"num_read_kb":702832,"num_write":195849,"num_write_kb":783390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1604268032,"data_stored":4259475494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40191,"ondisk_log_size":40191,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":3350,"num_objects":103,"num_object_clones":0,"num_object_copies":206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":103,"num_whiteouts":0,"num_read":11797,"num_read_kb":9931,"num_write":363,"num_write_kb":363,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":655360,"data_stored":6700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5753,"ondisk_log_size":5753,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":7460,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738470,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1564448,"kb_used_data":783192,"kb_used_omap":1,"kb_used_meta":781246,"kb_avail":92807392,"statfs":{"total":96636764160,"available":95034769408,"internally_reserved":0,"allocated":801988608,"data_stored":2126093800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":799996358},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,4,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 17:41:48 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.39400000000000002,"15min":0.46800000000000003},"min":{"1min":0.158,"5min":0.158,"15min":0.14299999999999999},"max":{"1min":1.8500000000000001,"5min":4.4640000000000004,"15min":4.4640000000000004},"last":0.29299999999999998},{"interface":"front","average":{"1min":0.41999999999999998,"5min":0.38,"15min":0.48899999999999999},"min":{"1min":0.19600000000000001,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":1.901,"5min":1.9850000000000001,"15min":1.9850000000000001},"last":0.25800000000000001}]},{"osd":1,"last update":"Tue Oct 3 17:41:48 2023","interfaces":[{"interface":"back","average":{"1min":0.29699999999999999,"5min":0.39000000000000001,"15min":0.49099999999999999},"min":{"1min":0.17599999999999999,"5min":0.17599999999999999,"15min":0.16600000000000001},"max":{"1min":0.69999999999999996,"5min":4.4260000000000002,"15min":4.4260000000000002},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.318,"5min":0.35399999999999998,"15min":0.50800000000000001},"min":{"1min":0.223,"5min":0.192,"15min":0.192},"max":{"1min":0.48299999999999998,"5min":2.0579999999999998,"15min":2.0579999999999998},"last":0.38}]}]},{"osd":1,"up_from":8,"seq":34359738468,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":819948,"kb_used_data":387748,"kb_used_omap":1,"kb_used_meta":432190,"kb_avail":93551892,"statfs":{"total":96636764160,"available":95797137408,"internally_reserved":0,"allocated":397053952,"data_stored":1049409794,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":442563014},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,5,2,11,3,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 17:42:02 2023","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":0.36699999999999999,"15min":0.45600000000000002},"min":{"1min":0.17499999999999999,"5min":0.161,"15min":0.161},"max":{"1min":1.2809999999999999,"5min":4.1879999999999997,"15min":4.1879999999999997},"last":0.247},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.33700000000000002,"15min":0.46300000000000002},"min":{"1min":0.17599999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":2.8399999999999999,"5min":2.8399999999999999,"15min":2.8399999999999999},"last":0.27400000000000002}]},{"osd":2,"last update":"Tue Oct 3 17:42:02 2023","interfaces":[{"interface":"back","average":{"1min":0.33300000000000002,"5min":0.35999999999999999,"15min":0.47199999999999998},"min":{"1min":0.20100000000000001,"5min":0.17799999999999999,"15min":0.159},"max":{"1min":0.65100000000000002,"5min":3.0390000000000001,"15min":3.0390000000000001},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.29999999999999999,"5min":0.36199999999999999,"15min":0.49299999999999999},"min":{"1min":0.216,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":0.51300000000000001,"5min":3.0699999999999998,"15min":3.0699999999999998},"last":0.34999999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738468,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":802088,"kb_used_data":396256,"kb_used_omap":1,"kb_used_meta":405822,"kb_avail":93569752,"statfs":{"total":96636764160,"available":95815426048,"internally_reserved":0,"allocated":405766144,"data_stored":1077449939,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":415562182},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,4,5,0,5,2,8,11],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 17:41:48 2023","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":0.46200000000000002,"15min":0.44700000000000001},"min":{"1min":0.16500000000000001,"5min":0.154,"15min":0.154},"max":{"1min":5.7649999999999997,"5min":5.7649999999999997,"15min":5.7649999999999997},"last":0.224},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.40300000000000002,"15min":0.45000000000000001},"min":{"1min":0.19,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":1.6259999999999999,"5min":3.1509999999999998,"15min":3.1509999999999998},"last":0.28199999999999997}]},{"osd":2,"last update":"Tue Oct 3 17:41:48 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.39800000000000002,"15min":0.51400000000000001},"min":{"1min":0.20699999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.66000000000000003,"5min":2.823,"15min":2.823},"last":0.25700000000000001},{"interface":"front","average":{"1min":0.34399999999999997,"5min":0.42599999999999999,"15min":0.503},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":0.57699999999999996,"5min":2.9239999999999999,"15min":2.9239999999999999},"last":0.29699999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":225280,"data_stored":2298,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":3350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":1052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":405282816,"data_stored":1078214656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":396914688,"data_stored":1052012563,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":802070528,"data_stored":2129248275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-03T17:42:34.831 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-03T17:34:47.281095+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=34, tm_sec=47, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=40, tm_sec=6, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T17:42:34.831 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-03T17:34:47.281095+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=34, tm_sec=47, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=40, tm_sec=6, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T17:42:34.832 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-03T17:34:47.281095+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=34, tm_sec=47, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=40, tm_sec=6, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T17:42:34.832 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-03T17:34:47.281095+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=34, tm_sec=47, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=40, tm_sec=6, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T17:42:34.832 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.3 2023-10-03T17:42:35.285 INFO:teuthology.orchestra.run.smithi111.stderr:instructing pg 3.3 on osd.0 to deep-scrub 2023-10-03T17:42:35.313 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.0 2023-10-03T17:42:35.781 INFO:teuthology.orchestra.run.smithi111.stderr:instructing pg 3.0 on osd.1 to deep-scrub 2023-10-03T17:42:35.807 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.1 2023-10-03T17:42:36.373 INFO:teuthology.orchestra.run.smithi111.stderr:instructing pg 3.1 on osd.0 to deep-scrub 2023-10-03T17:42:36.408 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.2 2023-10-03T17:42:36.948 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:42:36.946+0000 7f09e4851640 -1 reset not still connected to 0x557d76621040 2023-10-03T17:42:36.951 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:42:36.950+0000 7f09e4851640 -1 reset not still connected to 0x557d7664f380 2023-10-03T17:42:36.964 INFO:teuthology.orchestra.run.smithi111.stderr:instructing pg 3.2 on osd.1 to deep-scrub 2023-10-03T17:42:36.998 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T17:42:38.841 INFO:tasks.ceph.osd.0.smithi111.stderr:2023-10-03T17:42:38.842+0000 7ffbac289640 -1 reset not still connected to 0x5563a7ff65b0 2023-10-03T17:42:38.842 INFO:tasks.ceph.osd.0.smithi111.stderr:2023-10-03T17:42:38.842+0000 7ffbac289640 -1 reset not still connected to 0x5563a7ff68f0 2023-10-03T17:42:38.842 INFO:tasks.ceph.osd.0.smithi111.stderr:2023-10-03T17:42:38.842+0000 7ffbac289640 -1 reset not still connected to 0x5563a838e1a0 2023-10-03T17:42:38.842 INFO:tasks.ceph.osd.0.smithi111.stderr:2023-10-03T17:42:38.842+0000 7ffbac289640 -1 reset not still connected to 0x5563a851aea0 2023-10-03T17:42:55.299 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:42:55.297+0000 7f09e4851640 -1 reset not still connected to 0x557d75f2b2b0 2023-10-03T17:42:55.299 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:42:55.297+0000 7f09e4851640 -1 reset not still connected to 0x557d76253d40 2023-10-03T17:42:55.299 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:42:55.297+0000 7f09e4851640 -1 reset not still connected to 0x557d76621040 2023-10-03T17:42:55.299 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:42:55.297+0000 7f09e4851640 -1 reset not still connected to 0x557d7664e750 2023-10-03T17:42:55.300 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:42:55.297+0000 7f09e4851640 -1 reset not still connected to 0x557d7664ea90 2023-10-03T17:42:55.300 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:42:55.297+0000 7f09e4851640 -1 reset not still connected to 0x557d7664f380 2023-10-03T17:42:55.300 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:42:55.297+0000 7f09e4851640 -1 reset not still connected to 0x557d7664f930 2023-10-03T17:42:55.300 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:42:55.297+0000 7f09e4851640 -1 reset not still connected to 0x557d76681ba0 2023-10-03T17:42:55.300 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:42:55.297+0000 7f09e4851640 -1 reset not still connected to 0x557d766a40d0 2023-10-03T17:42:55.300 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:42:55.297+0000 7f09e4851640 -1 reset not still connected to 0x557d766b56c0 2023-10-03T17:42:55.300 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:42:55.297+0000 7f09e4851640 -1 reset not still connected to 0x557d76eca270 2023-10-03T17:42:57.000 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T17:42:57.481 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-03T17:42:57.481 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-10-03T17:42:57.525 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":299,"stamp":"2023-10-03T17:42:56.064527+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":21100797769,"num_objects":15347,"num_object_clones":10121,"num_object_copies":30694,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15347,"num_whiteouts":0,"num_read":211909,"num_read_kb":744326,"num_write":204531,"num_write_kb":818375,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":46503,"ondisk_log_size":46503,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3288120,"kb_used_data":1632736,"kb_used_omap":4,"kb_used_meta":1655355,"kb_avail":279827400,"statfs":{"total":289910292480,"available":286543257600,"internally_reserved":0,"allocated":1671921664,"data_stored":4430138301,"data_compressed":7460,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4782,"internal_metadata":1695083858},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,4,2,4,6,6,22,12,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":37195776,"num_objects":85,"num_object_clones":85,"num_object_copies":170,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":85,"num_whiteouts":0,"num_read":4006,"num_read_kb":15556,"num_write":4105,"num_write_kb":16420,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001993"},"pg_stats":[{"pgid":"2.7","version":"51'1106","reported_seq":1910,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-10-03T17:42:52.726099+0000","last_change":"2023-10-03T17:40:17.407011+0000","last_active":"2023-10-03T17:42:52.726099+0000","last_peered":"2023-10-03T17:42:52.726099+0000","last_clean":"2023-10-03T17:42:52.726099+0000","last_became_active":"2023-10-03T17:34:42.276624+0000","last_became_peered":"2023-10-03T17:34:42.276624+0000","last_unstale":"2023-10-03T17:42:52.726099+0000","last_undegraded":"2023-10-03T17:42:52.726099+0000","last_fullsized":"2023-10-03T17:42:52.726099+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'673","last_scrub_stamp":"2023-10-03T17:40:17.406813+0000","last_deep_scrub":"46'673","last_deep_scrub_stamp":"2023-10-03T17:40:17.406813+0000","last_clean_scrub_stamp":"2023-10-03T17:40:17.406813+0000","objects_scrubbed":7,"log_size":1106,"log_dups_size":0,"ondisk_log_size":1106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T21:38:35.056889+0000","scrub_duration":0.013778076,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1599,"num_read_kb":1329,"num_write":40,"num_write_kb":40,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":903,"num_omap_keys":30,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"51'1860","reported_seq":3636,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-10-03T17:42:52.726961+0000","last_change":"2023-10-03T17:40:13.376519+0000","last_active":"2023-10-03T17:42:52.726961+0000","last_peered":"2023-10-03T17:42:52.726961+0000","last_clean":"2023-10-03T17:42:52.726961+0000","last_became_active":"2023-10-03T17:34:42.271201+0000","last_became_peered":"2023-10-03T17:34:42.271201+0000","last_unstale":"2023-10-03T17:42:52.726961+0000","last_undegraded":"2023-10-03T17:42:52.726961+0000","last_fullsized":"2023-10-03T17:42:52.726961+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'1129","last_scrub_stamp":"2023-10-03T17:40:13.376332+0000","last_deep_scrub":"46'1129","last_deep_scrub_stamp":"2023-10-03T17:40:13.376332+0000","last_clean_scrub_stamp":"2023-10-03T17:40:13.376332+0000","objects_scrubbed":11,"log_size":1860,"log_dups_size":0,"ondisk_log_size":1860,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T18:59:17.088585+0000","scrub_duration":0.015998675,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":522,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":3404,"num_read_kb":2969,"num_write":104,"num_write_kb":104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2912,"num_omap_keys":93,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"50'21","reported_seq":101,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-10-03T17:42:01.409608+0000","last_change":"2023-10-03T17:40:25.496010+0000","last_active":"2023-10-03T17:42:01.409608+0000","last_peered":"2023-10-03T17:42:01.409608+0000","last_clean":"2023-10-03T17:42:01.409608+0000","last_became_active":"2023-10-03T17:34:42.276941+0000","last_became_peered":"2023-10-03T17:34:42.276941+0000","last_unstale":"2023-10-03T17:42:01.409608+0000","last_undegraded":"2023-10-03T17:42:01.409608+0000","last_fullsized":"2023-10-03T17:42:01.409608+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'14","last_scrub_stamp":"2023-10-03T17:40:25.495827+0000","last_deep_scrub":"46'14","last_deep_scrub_stamp":"2023-10-03T17:40:25.495827+0000","last_clean_scrub_stamp":"2023-10-03T17:40:25.495827+0000","objects_scrubbed":7,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:05:44.939663+0000","scrub_duration":0.0133078,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":604,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":14,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"51'1490","reported_seq":2809,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-10-03T17:42:52.726340+0000","last_change":"2023-10-03T17:40:26.513902+0000","last_active":"2023-10-03T17:42:52.726340+0000","last_peered":"2023-10-03T17:42:52.726340+0000","last_clean":"2023-10-03T17:42:52.726340+0000","last_became_active":"2023-10-03T17:34:42.276459+0000","last_became_peered":"2023-10-03T17:34:42.276459+0000","last_unstale":"2023-10-03T17:42:52.726340+0000","last_undegraded":"2023-10-03T17:42:52.726340+0000","last_fullsized":"2023-10-03T17:42:52.726340+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'964","last_scrub_stamp":"2023-10-03T17:40:26.513729+0000","last_deep_scrub":"46'964","last_deep_scrub_stamp":"2023-10-03T17:40:26.513729+0000","last_clean_scrub_stamp":"2023-10-03T17:40:26.513729+0000","objects_scrubbed":10,"log_size":1490,"log_dups_size":0,"ondisk_log_size":1490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:41:33.182751+0000","scrub_duration":0.018190280999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":2434,"num_read_kb":2047,"num_write":66,"num_write_kb":66,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1731,"num_omap_keys":80,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"51'55727","reported_seq":104936,"reported_epoch":51,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T17:42:53.927044+0000","last_change":"2023-10-03T17:40:11.716583+0000","last_active":"2023-10-03T17:42:53.927044+0000","last_peered":"2023-10-03T17:42:53.927044+0000","last_clean":"2023-10-03T17:42:53.927044+0000","last_became_active":"2023-10-03T17:34:48.310701+0000","last_became_peered":"2023-10-03T17:34:48.310701+0000","last_unstale":"2023-10-03T17:42:53.927044+0000","last_undegraded":"2023-10-03T17:42:53.927044+0000","last_fullsized":"2023-10-03T17:42:53.927044+0000","mapping_epoch":16,"log_start":"48'45627","ondisk_log_start":"48'45627","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_clean_scrub_stamp":"2023-10-03T17:34:47.281095+0000","objects_scrubbed":1435,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 162s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040768800000000002,"stat_sum":{"num_bytes":5434339328,"num_objects":3933,"num_object_clones":2613,"num_object_copies":7866,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3933,"num_whiteouts":0,"num_read":51582,"num_read_kb":189264,"num_write":52594,"num_write_kb":210376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"51'1437","reported_seq":2501,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-10-03T17:42:53.345461+0000","last_change":"2023-10-03T17:40:08.675119+0000","last_active":"2023-10-03T17:42:53.345461+0000","last_peered":"2023-10-03T17:42:53.345461+0000","last_clean":"2023-10-03T17:42:53.345461+0000","last_became_active":"2023-10-03T17:34:42.270625+0000","last_became_peered":"2023-10-03T17:34:42.270625+0000","last_unstale":"2023-10-03T17:42:53.345461+0000","last_undegraded":"2023-10-03T17:42:53.345461+0000","last_fullsized":"2023-10-03T17:42:53.345461+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'841","last_scrub_stamp":"2023-10-03T17:40:08.674910+0000","last_deep_scrub":"46'841","last_deep_scrub_stamp":"2023-10-03T17:40:08.674910+0000","last_clean_scrub_stamp":"2023-10-03T17:40:08.674910+0000","objects_scrubbed":11,"log_size":1437,"log_dups_size":0,"ondisk_log_size":1437,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T21:05:42.790614+0000","scrub_duration":0.039189555000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":605,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":2147,"num_read_kb":1790,"num_write":74,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2564,"num_omap_keys":60,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"51'51722","reported_seq":98044,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-10-03T17:42:53.824947+0000","last_change":"2023-10-03T17:42:37.972054+0000","last_active":"2023-10-03T17:42:53.824947+0000","last_peered":"2023-10-03T17:42:53.824947+0000","last_clean":"2023-10-03T17:42:53.824947+0000","last_became_active":"2023-10-03T17:34:48.310555+0000","last_became_peered":"2023-10-03T17:34:48.310555+0000","last_unstale":"2023-10-03T17:42:53.824947+0000","last_undegraded":"2023-10-03T17:42:53.824947+0000","last_fullsized":"2023-10-03T17:42:53.824947+0000","mapping_epoch":16,"log_start":"48'41622","ondisk_log_start":"48'41622","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_clean_scrub_stamp":"2023-10-03T17:34:47.281095+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086289699999999995,"stat_sum":{"num_bytes":5121372160,"num_objects":3693,"num_object_clones":2451,"num_object_copies":7386,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3693,"num_whiteouts":0,"num_read":48650,"num_read_kb":177992,"num_write":49271,"num_write_kb":197084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"50'66","reported_seq":857,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-10-03T17:42:52.732609+0000","last_change":"2023-10-03T17:40:17.641493+0000","last_active":"2023-10-03T17:42:52.732609+0000","last_peered":"2023-10-03T17:42:52.732609+0000","last_clean":"2023-10-03T17:42:52.732609+0000","last_became_active":"2023-10-03T17:34:42.278196+0000","last_became_peered":"2023-10-03T17:34:42.278196+0000","last_unstale":"2023-10-03T17:42:52.732609+0000","last_undegraded":"2023-10-03T17:42:52.732609+0000","last_fullsized":"2023-10-03T17:42:52.732609+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'30","last_scrub_stamp":"2023-10-03T17:40:17.640969+0000","last_deep_scrub":"46'30","last_deep_scrub_stamp":"2023-10-03T17:40:17.640969+0000","last_clean_scrub_stamp":"2023-10-03T17:40:17.640969+0000","objects_scrubbed":6,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:04:02.954896+0000","scrub_duration":0.071054565,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":318,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":1585,"num_read_kb":1324,"num_write":37,"num_write_kb":37,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":903,"num_omap_keys":30,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"51'53812","reported_seq":101186,"reported_epoch":51,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T17:42:53.914992+0000","last_change":"2023-10-03T17:40:10.712199+0000","last_active":"2023-10-03T17:42:53.914992+0000","last_peered":"2023-10-03T17:42:53.914992+0000","last_clean":"2023-10-03T17:42:53.914992+0000","last_became_active":"2023-10-03T17:34:48.310912+0000","last_became_peered":"2023-10-03T17:34:48.310912+0000","last_unstale":"2023-10-03T17:42:53.914992+0000","last_undegraded":"2023-10-03T17:42:53.914992+0000","last_fullsized":"2023-10-03T17:42:53.914992+0000","mapping_epoch":16,"log_start":"48'43726","ondisk_log_start":"48'43726","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_clean_scrub_stamp":"2023-10-03T17:34:47.281095+0000","objects_scrubbed":1932,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 163s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045661900000000001,"stat_sum":{"num_bytes":5238013952,"num_objects":3778,"num_object_clones":2507,"num_object_copies":7556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3778,"num_whiteouts":0,"num_read":49622,"num_read_kb":182888,"num_write":50547,"num_write_kb":202188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"49'35","reported_seq":358,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-10-03T17:42:47.609171+0000","last_change":"2023-10-03T17:40:10.708344+0000","last_active":"2023-10-03T17:42:47.609171+0000","last_peered":"2023-10-03T17:42:47.609171+0000","last_clean":"2023-10-03T17:42:47.609171+0000","last_became_active":"2023-10-03T17:34:42.272306+0000","last_became_peered":"2023-10-03T17:34:42.272306+0000","last_unstale":"2023-10-03T17:42:47.609171+0000","last_undegraded":"2023-10-03T17:42:47.609171+0000","last_fullsized":"2023-10-03T17:42:47.609171+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'20","last_scrub_stamp":"2023-10-03T17:40:10.708060+0000","last_deep_scrub":"46'20","last_deep_scrub_stamp":"2023-10-03T17:40:10.708060+0000","last_clean_scrub_stamp":"2023-10-03T17:40:10.708060+0000","objects_scrubbed":7,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:03:09.097252+0000","scrub_duration":0.042279491000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":451,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":563,"num_read_kb":452,"num_write":21,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":301,"num_omap_keys":10,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"51'54832","reported_seq":102658,"reported_epoch":51,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T17:42:53.819600+0000","last_change":"2023-10-03T17:40:29.562114+0000","last_active":"2023-10-03T17:42:53.819600+0000","last_peered":"2023-10-03T17:42:53.819600+0000","last_clean":"2023-10-03T17:42:53.819600+0000","last_became_active":"2023-10-03T17:34:48.310612+0000","last_became_peered":"2023-10-03T17:34:48.310612+0000","last_unstale":"2023-10-03T17:42:53.819600+0000","last_undegraded":"2023-10-03T17:42:53.819600+0000","last_fullsized":"2023-10-03T17:42:53.819600+0000","mapping_epoch":16,"log_start":"48'44740","ondisk_log_start":"48'44740","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_clean_scrub_stamp":"2023-10-03T17:34:47.281095+0000","objects_scrubbed":1523,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 144s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022375120000000001,"stat_sum":{"num_bytes":5306478611,"num_objects":3838,"num_object_clones":2550,"num_object_copies":7676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3838,"num_whiteouts":0,"num_read":50152,"num_read_kb":184160,"num_write":51619,"num_write_kb":206470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"50'11","reported_seq":84,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-10-03T17:42:01.309426+0000","last_change":"2023-10-03T17:40:24.472734+0000","last_active":"2023-10-03T17:42:01.309426+0000","last_peered":"2023-10-03T17:42:01.309426+0000","last_clean":"2023-10-03T17:42:01.309426+0000","last_became_active":"2023-10-03T17:34:42.270194+0000","last_became_peered":"2023-10-03T17:34:42.270194+0000","last_unstale":"2023-10-03T17:42:01.309426+0000","last_undegraded":"2023-10-03T17:42:01.309426+0000","last_fullsized":"2023-10-03T17:42:01.309426+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'8","last_scrub_stamp":"2023-10-03T17:40:24.472573+0000","last_deep_scrub":"46'8","last_deep_scrub_stamp":"2023-10-03T17:40:24.472573+0000","last_clean_scrub_stamp":"2023-10-03T17:40:24.472573+0000","objects_scrubbed":4,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T18:28:04.735033+0000","scrub_duration":0.024990637999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":283,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":8,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"46'99","reported_seq":192,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-10-03T17:42:02.262973+0000","last_change":"2023-10-03T17:40:10.452341+0000","last_active":"2023-10-03T17:42:02.262973+0000","last_peered":"2023-10-03T17:42:02.262973+0000","last_clean":"2023-10-03T17:42:02.262973+0000","last_became_active":"2023-10-03T17:34:39.502393+0000","last_became_peered":"2023-10-03T17:34:39.502393+0000","last_unstale":"2023-10-03T17:42:02.262973+0000","last_undegraded":"2023-10-03T17:42:02.262973+0000","last_fullsized":"2023-10-03T17:42:02.262973+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'99","last_scrub_stamp":"2023-10-03T17:40:10.449099+0000","last_deep_scrub":"46'99","last_deep_scrub_stamp":"2023-10-03T17:40:10.449099+0000","last_clean_scrub_stamp":"2023-10-03T17:40:10.449099+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T20:18:04.958666+0000","scrub_duration":0.043225012,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":21100204051,"num_objects":15242,"num_object_clones":10121,"num_object_copies":30484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15242,"num_whiteouts":0,"num_read":200006,"num_read_kb":734304,"num_write":204031,"num_write_kb":816118,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1671225344,"data_stored":4430757926,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40378,"ondisk_log_size":40378,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":3350,"num_objects":103,"num_object_clones":0,"num_object_copies":206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":103,"num_whiteouts":0,"num_read":11797,"num_read_kb":9931,"num_write":363,"num_write_kb":363,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":655360,"data_stored":6700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6026,"ondisk_log_size":6026,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":7460,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738474,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1610896,"kb_used_data":815752,"kb_used_omap":1,"kb_used_meta":795134,"kb_avail":92760944,"statfs":{"total":96636764160,"available":94987206656,"internally_reserved":0,"allocated":835330048,"data_stored":2214145512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":814217670},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,1,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 17:42:49 2023","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.44800000000000001,"15min":0.48299999999999998},"min":{"1min":0.17499999999999999,"5min":0.158,"15min":0.14299999999999999},"max":{"1min":4.1539999999999999,"5min":4.4640000000000004,"15min":4.4640000000000004},"last":0.21299999999999999},{"interface":"front","average":{"1min":0.73899999999999999,"5min":0.46200000000000002,"15min":0.501},"min":{"1min":0.17100000000000001,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":4.4180000000000001,"5min":4.4180000000000001,"15min":4.4180000000000001},"last":0.40999999999999998}]},{"osd":1,"last update":"Tue Oct 3 17:42:49 2023","interfaces":[{"interface":"back","average":{"1min":1.022,"5min":0.502,"15min":0.52200000000000002},"min":{"1min":0.23100000000000001,"5min":0.17599999999999999,"15min":0.16600000000000001},"max":{"1min":5.2359999999999998,"5min":5.2359999999999998,"15min":5.2359999999999998},"last":0.26400000000000001},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.439,"15min":0.51700000000000002},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.192},"max":{"1min":3.9740000000000002,"5min":3.9740000000000002,"15min":3.9740000000000002},"last":0.44500000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738472,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":836124,"kb_used_data":403924,"kb_used_omap":1,"kb_used_meta":432190,"kb_avail":93535716,"statfs":{"total":96636764160,"available":95780573184,"internally_reserved":0,"allocated":413618176,"data_stored":1095694594,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":442563014},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,3,0,1,2,3,7,7,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 17:42:02 2023","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":0.36699999999999999,"15min":0.45600000000000002},"min":{"1min":0.17499999999999999,"5min":0.161,"15min":0.161},"max":{"1min":1.2809999999999999,"5min":4.1879999999999997,"15min":4.1879999999999997},"last":0.217},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.33700000000000002,"15min":0.46300000000000002},"min":{"1min":0.17599999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":2.8399999999999999,"5min":2.8399999999999999,"15min":2.8399999999999999},"last":0.26600000000000001}]},{"osd":2,"last update":"Tue Oct 3 17:42:02 2023","interfaces":[{"interface":"back","average":{"1min":0.33300000000000002,"5min":0.35999999999999999,"15min":0.47199999999999998},"min":{"1min":0.20100000000000001,"5min":0.17799999999999999,"15min":0.159},"max":{"1min":0.65100000000000002,"5min":3.0390000000000001,"15min":3.0390000000000001},"last":0.29699999999999999},{"interface":"front","average":{"1min":0.29999999999999999,"5min":0.36199999999999999,"15min":0.49299999999999999},"min":{"1min":0.216,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":0.51300000000000001,"5min":3.0699999999999998,"15min":3.0699999999999998},"last":0.253}]}]},{"osd":0,"up_from":8,"seq":34359738472,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":841100,"kb_used_data":413060,"kb_used_omap":1,"kb_used_meta":428030,"kb_avail":93530740,"statfs":{"total":96636764160,"available":95775477760,"internally_reserved":0,"allocated":422973440,"data_stored":1120298195,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":438303174},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,2,2,2,15,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 17:42:49 2023","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.47399999999999998,"15min":0.443},"min":{"1min":0.191,"5min":0.154,"15min":0.154},"max":{"1min":2.1349999999999998,"5min":5.7649999999999997,"15min":5.7649999999999997},"last":0.251},{"interface":"front","average":{"1min":1.1419999999999999,"5min":0.54300000000000004,"15min":0.49399999999999999},"min":{"1min":0.20499999999999999,"5min":0.19,"15min":0.17699999999999999},"max":{"1min":9.4770000000000003,"5min":9.4770000000000003,"15min":9.4770000000000003},"last":0.33200000000000002}]},{"osd":2,"last update":"Tue Oct 3 17:42:49 2023","interfaces":[{"interface":"back","average":{"1min":1.018,"5min":0.53600000000000003,"15min":0.54300000000000004},"min":{"1min":0.24199999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":9.4149999999999991,"5min":9.4149999999999991,"15min":9.4149999999999991},"last":0.31},{"interface":"front","average":{"1min":0.38600000000000001,"5min":0.42899999999999999,"15min":0.49099999999999999},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":1.5509999999999999,"5min":2.9239999999999999,"15min":2.9239999999999999},"last":0.27300000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":225280,"data_stored":2298,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":3350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":1052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":422469632,"data_stored":1119866880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":413249536,"data_stored":1095700499,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":835506176,"data_stored":2215190547,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-03T17:42:57.527 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-03T17:34:47.281095+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=34, tm_sec=47, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=40, tm_sec=6, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T17:42:57.527 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-03T17:34:47.281095+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=34, tm_sec=47, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=40, tm_sec=6, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T17:42:57.527 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-03T17:34:47.281095+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=34, tm_sec=47, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=40, tm_sec=6, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T17:42:57.528 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-03T17:34:47.281095+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=34, tm_sec=47, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=40, tm_sec=6, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T17:42:57.528 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T17:43:17.529 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T17:43:18.032 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-03T17:43:18.033 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-10-03T17:43:18.066 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":311,"stamp":"2023-10-03T17:43:16.068138+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":21171401532,"num_objects":18483,"num_object_clones":13237,"num_object_copies":36966,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18483,"num_whiteouts":0,"num_read":223128,"num_read_kb":777286,"num_write":212222,"num_write_kb":848899,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":46633,"ondisk_log_size":46633,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3467096,"kb_used_data":1693756,"kb_used_omap":4,"kb_used_meta":1773307,"kb_avail":279648424,"statfs":{"total":289910292480,"available":286359986176,"internally_reserved":0,"allocated":1734406144,"data_stored":5492834449,"data_compressed":7460,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4782,"internal_metadata":1815866706},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,8,9,12,21,12,1,3],"upper_bound":1024},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":31330304,"num_objects":2184,"num_object_clones":2184,"num_object_copies":4368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2184,"num_whiteouts":0,"num_read":5119,"num_read_kb":15915,"num_write":3663,"num_write_kb":14652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002381"},"pg_stats":[{"pgid":"2.7","version":"53'1153","reported_seq":2085,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-10-03T17:43:13.206252+0000","last_change":"2023-10-03T17:40:17.407011+0000","last_active":"2023-10-03T17:43:13.206252+0000","last_peered":"2023-10-03T17:43:13.206252+0000","last_clean":"2023-10-03T17:43:13.206252+0000","last_became_active":"2023-10-03T17:34:42.276624+0000","last_became_peered":"2023-10-03T17:34:42.276624+0000","last_unstale":"2023-10-03T17:43:13.206252+0000","last_undegraded":"2023-10-03T17:43:13.206252+0000","last_fullsized":"2023-10-03T17:43:13.206252+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'673","last_scrub_stamp":"2023-10-03T17:40:17.406813+0000","last_deep_scrub":"46'673","last_deep_scrub_stamp":"2023-10-03T17:40:17.406813+0000","last_clean_scrub_stamp":"2023-10-03T17:40:17.406813+0000","objects_scrubbed":7,"log_size":1153,"log_dups_size":0,"ondisk_log_size":1153,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T21:38:35.056889+0000","scrub_duration":0.013778076,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":537,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":2079,"num_read_kb":1716,"num_write":51,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":903,"num_omap_keys":30,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"53'1936","reported_seq":4046,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-10-03T17:43:13.206730+0000","last_change":"2023-10-03T17:40:13.376519+0000","last_active":"2023-10-03T17:43:13.206730+0000","last_peered":"2023-10-03T17:43:13.206730+0000","last_clean":"2023-10-03T17:43:13.206730+0000","last_became_active":"2023-10-03T17:34:42.271201+0000","last_became_peered":"2023-10-03T17:34:42.271201+0000","last_unstale":"2023-10-03T17:43:13.206730+0000","last_undegraded":"2023-10-03T17:43:13.206730+0000","last_fullsized":"2023-10-03T17:43:13.206730+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'1129","last_scrub_stamp":"2023-10-03T17:40:13.376332+0000","last_deep_scrub":"46'1129","last_deep_scrub_stamp":"2023-10-03T17:40:13.376332+0000","last_clean_scrub_stamp":"2023-10-03T17:40:13.376332+0000","objects_scrubbed":11,"log_size":1936,"log_dups_size":0,"ondisk_log_size":1936,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T18:59:17.088585+0000","scrub_duration":0.015998675,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":572,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":4368,"num_read_kb":3777,"num_write":120,"num_write_kb":120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2912,"num_omap_keys":93,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"53'27","reported_seq":110,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-10-03T17:43:01.322917+0000","last_change":"2023-10-03T17:40:25.496010+0000","last_active":"2023-10-03T17:43:01.322917+0000","last_peered":"2023-10-03T17:43:01.322917+0000","last_clean":"2023-10-03T17:43:01.322917+0000","last_became_active":"2023-10-03T17:34:42.276941+0000","last_became_peered":"2023-10-03T17:34:42.276941+0000","last_unstale":"2023-10-03T17:43:01.322917+0000","last_undegraded":"2023-10-03T17:43:01.322917+0000","last_fullsized":"2023-10-03T17:43:01.322917+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'14","last_scrub_stamp":"2023-10-03T17:40:25.495827+0000","last_deep_scrub":"46'14","last_deep_scrub_stamp":"2023-10-03T17:40:25.495827+0000","last_clean_scrub_stamp":"2023-10-03T17:40:25.495827+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:05:44.939663+0000","scrub_duration":0.0133078,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":906,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"53'1552","reported_seq":3105,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-10-03T17:43:13.206413+0000","last_change":"2023-10-03T17:40:26.513902+0000","last_active":"2023-10-03T17:43:13.206413+0000","last_peered":"2023-10-03T17:43:13.206413+0000","last_clean":"2023-10-03T17:43:13.206413+0000","last_became_active":"2023-10-03T17:34:42.276459+0000","last_became_peered":"2023-10-03T17:34:42.276459+0000","last_unstale":"2023-10-03T17:43:13.206413+0000","last_undegraded":"2023-10-03T17:43:13.206413+0000","last_fullsized":"2023-10-03T17:43:13.206413+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'964","last_scrub_stamp":"2023-10-03T17:40:26.513729+0000","last_deep_scrub":"46'964","last_deep_scrub_stamp":"2023-10-03T17:40:26.513729+0000","last_clean_scrub_stamp":"2023-10-03T17:40:26.513729+0000","objects_scrubbed":10,"log_size":1552,"log_dups_size":0,"ondisk_log_size":1552,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:41:33.182751+0000","scrub_duration":0.018190280999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":3140,"num_read_kb":2628,"num_write":80,"num_write_kb":80,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1731,"num_omap_keys":80,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"53'58582","reported_seq":108997,"reported_epoch":53,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T17:43:13.974411+0000","last_change":"2023-10-03T17:40:11.716583+0000","last_active":"2023-10-03T17:43:13.974411+0000","last_peered":"2023-10-03T17:43:13.974411+0000","last_clean":"2023-10-03T17:43:13.974411+0000","last_became_active":"2023-10-03T17:34:48.310701+0000","last_became_peered":"2023-10-03T17:34:48.310701+0000","last_unstale":"2023-10-03T17:43:13.974411+0000","last_undegraded":"2023-10-03T17:43:13.974411+0000","last_fullsized":"2023-10-03T17:43:13.974411+0000","mapping_epoch":16,"log_start":"48'48539","ondisk_log_start":"48'48539","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_clean_scrub_stamp":"2023-10-03T17:34:47.281095+0000","objects_scrubbed":1598,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 182s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040768800000000002,"stat_sum":{"num_bytes":5454270464,"num_objects":4727,"num_object_clones":3407,"num_object_copies":9454,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4727,"num_whiteouts":0,"num_read":53564,"num_read_kb":196920,"num_write":54548,"num_write_kb":218192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"53'1486","reported_seq":2724,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-10-03T17:43:13.206747+0000","last_change":"2023-10-03T17:40:08.675119+0000","last_active":"2023-10-03T17:43:13.206747+0000","last_peered":"2023-10-03T17:43:13.206747+0000","last_clean":"2023-10-03T17:43:13.206747+0000","last_became_active":"2023-10-03T17:34:42.270625+0000","last_became_peered":"2023-10-03T17:34:42.270625+0000","last_unstale":"2023-10-03T17:43:13.206747+0000","last_undegraded":"2023-10-03T17:43:13.206747+0000","last_fullsized":"2023-10-03T17:43:13.206747+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'841","last_scrub_stamp":"2023-10-03T17:40:08.674910+0000","last_deep_scrub":"46'841","last_deep_scrub_stamp":"2023-10-03T17:40:08.674910+0000","last_clean_scrub_stamp":"2023-10-03T17:40:08.674910+0000","objects_scrubbed":11,"log_size":1486,"log_dups_size":0,"ondisk_log_size":1486,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T21:05:42.790614+0000","scrub_duration":0.039189555000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":706,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":2790,"num_read_kb":2310,"num_write":88,"num_write_kb":88,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2564,"num_omap_keys":60,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"53'54404","reported_seq":101867,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-10-03T17:43:13.956925+0000","last_change":"2023-10-03T17:43:04.438717+0000","last_active":"2023-10-03T17:43:13.956925+0000","last_peered":"2023-10-03T17:43:13.956925+0000","last_clean":"2023-10-03T17:43:13.956925+0000","last_became_active":"2023-10-03T17:34:48.310555+0000","last_became_peered":"2023-10-03T17:34:48.310555+0000","last_unstale":"2023-10-03T17:43:13.956925+0000","last_undegraded":"2023-10-03T17:43:13.956925+0000","last_fullsized":"2023-10-03T17:43:13.956925+0000","mapping_epoch":16,"log_start":"48'44328","ondisk_log_start":"48'44328","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_clean_scrub_stamp":"2023-10-03T17:34:47.281095+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086289699999999995,"stat_sum":{"num_bytes":5137076224,"num_objects":4471,"num_object_clones":3229,"num_object_copies":8942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4471,"num_whiteouts":0,"num_read":50560,"num_read_kb":185376,"num_write":51175,"num_write_kb":204700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"53'79","reported_seq":1001,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-10-03T17:43:13.695901+0000","last_change":"2023-10-03T17:40:17.641493+0000","last_active":"2023-10-03T17:43:13.695901+0000","last_peered":"2023-10-03T17:43:13.695901+0000","last_clean":"2023-10-03T17:43:13.695901+0000","last_became_active":"2023-10-03T17:34:42.278196+0000","last_became_peered":"2023-10-03T17:34:42.278196+0000","last_unstale":"2023-10-03T17:43:13.695901+0000","last_undegraded":"2023-10-03T17:43:13.695901+0000","last_fullsized":"2023-10-03T17:43:13.695901+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'30","last_scrub_stamp":"2023-10-03T17:40:17.640969+0000","last_deep_scrub":"46'30","last_deep_scrub_stamp":"2023-10-03T17:40:17.640969+0000","last_clean_scrub_stamp":"2023-10-03T17:40:17.640969+0000","objects_scrubbed":6,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:04:02.954896+0000","scrub_duration":0.071054565,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":2065,"num_read_kb":1711,"num_write":47,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":903,"num_omap_keys":30,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"53'56617","reported_seq":105274,"reported_epoch":53,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T17:43:13.970416+0000","last_change":"2023-10-03T17:40:10.712199+0000","last_active":"2023-10-03T17:43:13.970416+0000","last_peered":"2023-10-03T17:43:13.970416+0000","last_clean":"2023-10-03T17:43:13.970416+0000","last_became_active":"2023-10-03T17:34:48.310912+0000","last_became_peered":"2023-10-03T17:34:48.310912+0000","last_unstale":"2023-10-03T17:43:13.970416+0000","last_undegraded":"2023-10-03T17:43:13.970416+0000","last_fullsized":"2023-10-03T17:43:13.970416+0000","mapping_epoch":16,"log_start":"48'46536","ondisk_log_start":"48'46536","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_clean_scrub_stamp":"2023-10-03T17:34:47.281095+0000","objects_scrubbed":2162,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 183s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045661900000000001,"stat_sum":{"num_bytes":5255938048,"num_objects":4549,"num_object_clones":3278,"num_object_copies":9098,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4549,"num_whiteouts":0,"num_read":51661,"num_read_kb":190780,"num_write":52429,"num_write_kb":209716,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"53'43","reported_seq":410,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-10-03T17:43:08.289431+0000","last_change":"2023-10-03T17:40:10.708344+0000","last_active":"2023-10-03T17:43:08.289431+0000","last_peered":"2023-10-03T17:43:08.289431+0000","last_clean":"2023-10-03T17:43:08.289431+0000","last_became_active":"2023-10-03T17:34:42.272306+0000","last_became_peered":"2023-10-03T17:34:42.272306+0000","last_unstale":"2023-10-03T17:43:08.289431+0000","last_undegraded":"2023-10-03T17:43:08.289431+0000","last_fullsized":"2023-10-03T17:43:08.289431+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'20","last_scrub_stamp":"2023-10-03T17:40:10.708060+0000","last_deep_scrub":"46'20","last_deep_scrub_stamp":"2023-10-03T17:40:10.708060+0000","last_clean_scrub_stamp":"2023-10-03T17:40:10.708060+0000","objects_scrubbed":7,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:03:09.097252+0000","scrub_duration":0.042279491000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":653,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":723,"num_read_kb":581,"num_write":28,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":301,"num_omap_keys":10,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"53'57595","reported_seq":106516,"reported_epoch":53,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T17:43:13.979183+0000","last_change":"2023-10-03T17:40:29.562114+0000","last_active":"2023-10-03T17:43:13.979183+0000","last_peered":"2023-10-03T17:43:13.979183+0000","last_clean":"2023-10-03T17:43:13.979183+0000","last_became_active":"2023-10-03T17:34:48.310612+0000","last_became_peered":"2023-10-03T17:34:48.310612+0000","last_unstale":"2023-10-03T17:43:13.979183+0000","last_undegraded":"2023-10-03T17:43:13.979183+0000","last_fullsized":"2023-10-03T17:43:13.979183+0000","mapping_epoch":16,"log_start":"48'47550","ondisk_log_start":"48'47550","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_clean_scrub_stamp":"2023-10-03T17:34:47.281095+0000","objects_scrubbed":1703,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 164s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022375120000000001,"stat_sum":{"num_bytes":5323522067,"num_objects":4611,"num_object_clones":3323,"num_object_copies":9222,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4611,"num_whiteouts":0,"num_read":52007,"num_read_kb":191376,"num_write":53490,"num_write_kb":213954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"53'13","reported_seq":89,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-10-03T17:43:01.349428+0000","last_change":"2023-10-03T17:40:24.472734+0000","last_active":"2023-10-03T17:43:01.349428+0000","last_peered":"2023-10-03T17:43:01.349428+0000","last_clean":"2023-10-03T17:43:01.349428+0000","last_became_active":"2023-10-03T17:34:42.270194+0000","last_became_peered":"2023-10-03T17:34:42.270194+0000","last_unstale":"2023-10-03T17:43:01.349428+0000","last_undegraded":"2023-10-03T17:43:01.349428+0000","last_fullsized":"2023-10-03T17:43:01.349428+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'8","last_scrub_stamp":"2023-10-03T17:40:24.472573+0000","last_deep_scrub":"46'8","last_deep_scrub_stamp":"2023-10-03T17:40:24.472573+0000","last_clean_scrub_stamp":"2023-10-03T17:40:24.472573+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T18:28:04.735033+0000","scrub_duration":0.024990637999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":385,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":26,"num_read_kb":8,"num_write":9,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"46'99","reported_seq":194,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-10-03T17:43:00.295007+0000","last_change":"2023-10-03T17:40:10.452341+0000","last_active":"2023-10-03T17:43:00.295007+0000","last_peered":"2023-10-03T17:43:00.295007+0000","last_clean":"2023-10-03T17:43:00.295007+0000","last_became_active":"2023-10-03T17:34:39.502393+0000","last_became_peered":"2023-10-03T17:34:39.502393+0000","last_unstale":"2023-10-03T17:43:00.295007+0000","last_undegraded":"2023-10-03T17:43:00.295007+0000","last_fullsized":"2023-10-03T17:43:00.295007+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'99","last_scrub_stamp":"2023-10-03T17:40:10.449099+0000","last_deep_scrub":"46'99","last_deep_scrub_stamp":"2023-10-03T17:40:10.449099+0000","last_clean_scrub_stamp":"2023-10-03T17:40:10.449099+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T20:18:04.958666+0000","scrub_duration":0.043225012,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":21170806803,"num_objects":18358,"num_object_clones":13237,"num_object_copies":36716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18358,"num_whiteouts":0,"num_read":207792,"num_read_kb":764452,"num_write":211642,"num_write_kb":846562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1733730304,"data_stored":5518053414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40245,"ondisk_log_size":40245,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4361,"num_objects":123,"num_object_clones":0,"num_object_copies":246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":123,"num_whiteouts":0,"num_read":15230,"num_read_kb":12743,"num_write":443,"num_write_kb":443,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":819200,"data_stored":8722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6289,"ondisk_log_size":6289,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":7460,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738478,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1723252,"kb_used_data":846184,"kb_used_omap":1,"kb_used_meta":877054,"kb_avail":92648588,"statfs":{"total":96636764160,"available":94872154112,"internally_reserved":0,"allocated":866492416,"data_stored":2741923397,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":898103750},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,3,3,5,7,1,3],"upper_bound":1024},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 17:42:49 2023","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.44800000000000001,"15min":0.48299999999999998},"min":{"1min":0.17499999999999999,"5min":0.158,"15min":0.14299999999999999},"max":{"1min":4.1539999999999999,"5min":4.4640000000000004,"15min":4.4640000000000004},"last":0.254},{"interface":"front","average":{"1min":0.73899999999999999,"5min":0.46200000000000002,"15min":0.501},"min":{"1min":0.17100000000000001,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":4.4180000000000001,"5min":4.4180000000000001,"15min":4.4180000000000001},"last":0.27200000000000002}]},{"osd":1,"last update":"Tue Oct 3 17:42:49 2023","interfaces":[{"interface":"back","average":{"1min":1.022,"5min":0.502,"15min":0.52200000000000002},"min":{"1min":0.23100000000000001,"5min":0.17599999999999999,"15min":0.16600000000000001},"max":{"1min":5.2359999999999998,"5min":5.2359999999999998,"15min":5.2359999999999998},"last":0.29699999999999999},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.439,"15min":0.51700000000000002},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.192},"max":{"1min":3.9740000000000002,"5min":3.9740000000000002,"15min":3.9740000000000002},"last":0.318}]}]},{"osd":1,"up_from":8,"seq":34359738476,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":869412,"kb_used_data":419164,"kb_used_omap":1,"kb_used_meta":450238,"kb_avail":93502428,"statfs":{"total":96636764160,"available":95746486272,"internally_reserved":0,"allocated":429223936,"data_stored":1362516975,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":461044166},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,5,5,5,10],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 17:43:03 2023","interfaces":[{"interface":"back","average":{"1min":0.22600000000000001,"5min":0.31,"15min":0.436},"min":{"1min":0.17399999999999999,"5min":0.161,"15min":0.161},"max":{"1min":0.33700000000000002,"5min":1.8120000000000001,"15min":4.1879999999999997},"last":0.24099999999999999},{"interface":"front","average":{"1min":0.35699999999999998,"5min":0.33800000000000002,"15min":0.45000000000000001},"min":{"1min":0.20000000000000001,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":1.276,"5min":2.8399999999999999,"15min":2.8399999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Tue Oct 3 17:43:03 2023","interfaces":[{"interface":"back","average":{"1min":0.31,"5min":0.35399999999999998,"15min":0.45600000000000002},"min":{"1min":0.187,"5min":0.17799999999999999,"15min":0.159},"max":{"1min":0.41999999999999998,"5min":3.0390000000000001,"15min":3.0390000000000001},"last":0.21199999999999999},{"interface":"front","average":{"1min":0.28499999999999998,"5min":0.34899999999999998,"15min":0.47299999999999998},"min":{"1min":0.224,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":0.35099999999999998,"5min":3.0699999999999998,"15min":3.0699999999999998},"last":0.29599999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738476,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":874432,"kb_used_data":428408,"kb_used_omap":1,"kb_used_meta":446014,"kb_avail":93497408,"statfs":{"total":96636764160,"available":95741345792,"internally_reserved":0,"allocated":438689792,"data_stored":1388394077,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":456718790},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1,4,6,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 17:42:49 2023","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.47399999999999998,"15min":0.443},"min":{"1min":0.191,"5min":0.154,"15min":0.154},"max":{"1min":2.1349999999999998,"5min":5.7649999999999997,"15min":5.7649999999999997},"last":0.25800000000000001},{"interface":"front","average":{"1min":1.1419999999999999,"5min":0.54300000000000004,"15min":0.49399999999999999},"min":{"1min":0.20499999999999999,"5min":0.19,"15min":0.17699999999999999},"max":{"1min":9.4770000000000003,"5min":9.4770000000000003,"15min":9.4770000000000003},"last":0.27400000000000002}]},{"osd":2,"last update":"Tue Oct 3 17:42:49 2023","interfaces":[{"interface":"back","average":{"1min":1.018,"5min":0.53600000000000003,"15min":0.54300000000000004},"min":{"1min":0.24199999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":9.4149999999999991,"5min":9.4149999999999991,"15min":9.4149999999999991},"last":3.8239999999999998},{"interface":"front","average":{"1min":0.38600000000000001,"5min":0.42899999999999999,"15min":0.49099999999999999},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":1.5509999999999999,"5min":2.9239999999999999,"15min":2.9239999999999999},"last":3.8100000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":278528,"data_stored":2954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":409600,"data_stored":4361,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":438177792,"data_stored":1392168960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":428716032,"data_stored":1367253011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":866836480,"data_stored":2758631443,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-03T17:43:18.068 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-03T17:34:47.281095+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=34, tm_sec=47, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=40, tm_sec=6, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T17:43:18.069 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-03T17:34:47.281095+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=34, tm_sec=47, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=40, tm_sec=6, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T17:43:18.069 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-03T17:34:47.281095+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=34, tm_sec=47, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=40, tm_sec=6, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T17:43:18.069 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-03T17:34:47.281095+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=34, tm_sec=47, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=40, tm_sec=6, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T17:43:18.070 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T17:43:38.071 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T17:43:38.621 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-03T17:43:38.622 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-10-03T17:43:38.647 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":322,"stamp":"2023-10-03T17:43:38.075141+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":21230723900,"num_objects":20097,"num_object_clones":14851,"num_object_copies":40194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20097,"num_whiteouts":0,"num_read":231641,"num_read_kb":810326,"num_write":220045,"num_write_kb":880191,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":46773,"ondisk_log_size":46773,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3615696,"kb_used_data":1756344,"kb_used_omap":4,"kb_used_meta":1859323,"kb_avail":279499824,"statfs":{"total":289910292480,"available":286207819776,"internally_reserved":0,"allocated":1798496256,"data_stored":6107676817,"data_compressed":7460,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4782,"internal_metadata":1903947090},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,3,7,5,16,13,11],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":28258304,"num_objects":552,"num_object_clones":552,"num_object_copies":1104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":552,"num_whiteouts":0,"num_read":4315,"num_read_kb":16804,"num_write":3992,"num_write_kb":15968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001923"},"pg_stats":[{"pgid":"2.7","version":"53'1168","reported_seq":2113,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-10-03T17:43:33.690488+0000","last_change":"2023-10-03T17:40:17.407011+0000","last_active":"2023-10-03T17:43:33.690488+0000","last_peered":"2023-10-03T17:43:33.690488+0000","last_clean":"2023-10-03T17:43:33.690488+0000","last_became_active":"2023-10-03T17:34:42.276624+0000","last_became_peered":"2023-10-03T17:34:42.276624+0000","last_unstale":"2023-10-03T17:43:33.690488+0000","last_undegraded":"2023-10-03T17:43:33.690488+0000","last_fullsized":"2023-10-03T17:43:33.690488+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'673","last_scrub_stamp":"2023-10-03T17:40:17.406813+0000","last_deep_scrub":"46'673","last_deep_scrub_stamp":"2023-10-03T17:40:17.406813+0000","last_clean_scrub_stamp":"2023-10-03T17:40:17.406813+0000","objects_scrubbed":7,"log_size":1168,"log_dups_size":0,"ondisk_log_size":1168,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T21:38:35.056889+0000","scrub_duration":0.013778076,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":537,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":2079,"num_read_kb":1716,"num_write":51,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":903,"num_omap_keys":30,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"53'1961","reported_seq":4092,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-10-03T17:43:33.690861+0000","last_change":"2023-10-03T17:40:13.376519+0000","last_active":"2023-10-03T17:43:33.690861+0000","last_peered":"2023-10-03T17:43:33.690861+0000","last_clean":"2023-10-03T17:43:33.690861+0000","last_became_active":"2023-10-03T17:34:42.271201+0000","last_became_peered":"2023-10-03T17:34:42.271201+0000","last_unstale":"2023-10-03T17:43:33.690861+0000","last_undegraded":"2023-10-03T17:43:33.690861+0000","last_fullsized":"2023-10-03T17:43:33.690861+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'1129","last_scrub_stamp":"2023-10-03T17:40:13.376332+0000","last_deep_scrub":"46'1129","last_deep_scrub_stamp":"2023-10-03T17:40:13.376332+0000","last_clean_scrub_stamp":"2023-10-03T17:40:13.376332+0000","objects_scrubbed":11,"log_size":1961,"log_dups_size":0,"ondisk_log_size":1961,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T18:59:17.088585+0000","scrub_duration":0.015998675,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":572,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":4368,"num_read_kb":3777,"num_write":120,"num_write_kb":120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2912,"num_omap_keys":93,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"53'27","reported_seq":110,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-10-03T17:43:01.322917+0000","last_change":"2023-10-03T17:40:25.496010+0000","last_active":"2023-10-03T17:43:01.322917+0000","last_peered":"2023-10-03T17:43:01.322917+0000","last_clean":"2023-10-03T17:43:01.322917+0000","last_became_active":"2023-10-03T17:34:42.276941+0000","last_became_peered":"2023-10-03T17:34:42.276941+0000","last_unstale":"2023-10-03T17:43:01.322917+0000","last_undegraded":"2023-10-03T17:43:01.322917+0000","last_fullsized":"2023-10-03T17:43:01.322917+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'14","last_scrub_stamp":"2023-10-03T17:40:25.495827+0000","last_deep_scrub":"46'14","last_deep_scrub_stamp":"2023-10-03T17:40:25.495827+0000","last_clean_scrub_stamp":"2023-10-03T17:40:25.495827+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:05:44.939663+0000","scrub_duration":0.0133078,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":906,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"53'1572","reported_seq":3141,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-10-03T17:43:33.690611+0000","last_change":"2023-10-03T17:40:26.513902+0000","last_active":"2023-10-03T17:43:33.690611+0000","last_peered":"2023-10-03T17:43:33.690611+0000","last_clean":"2023-10-03T17:43:33.690611+0000","last_became_active":"2023-10-03T17:34:42.276459+0000","last_became_peered":"2023-10-03T17:34:42.276459+0000","last_unstale":"2023-10-03T17:43:33.690611+0000","last_undegraded":"2023-10-03T17:43:33.690611+0000","last_fullsized":"2023-10-03T17:43:33.690611+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'964","last_scrub_stamp":"2023-10-03T17:40:26.513729+0000","last_deep_scrub":"46'964","last_deep_scrub_stamp":"2023-10-03T17:40:26.513729+0000","last_clean_scrub_stamp":"2023-10-03T17:40:26.513729+0000","objects_scrubbed":10,"log_size":1572,"log_dups_size":0,"ondisk_log_size":1572,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:41:33.182751+0000","scrub_duration":0.018190280999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":3140,"num_read_kb":2628,"num_write":80,"num_write_kb":80,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1731,"num_omap_keys":80,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"53'61160","reported_seq":113312,"reported_epoch":53,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T17:43:34.128201+0000","last_change":"2023-10-03T17:40:11.716583+0000","last_active":"2023-10-03T17:43:34.128201+0000","last_peered":"2023-10-03T17:43:34.128201+0000","last_clean":"2023-10-03T17:43:34.128201+0000","last_became_active":"2023-10-03T17:34:48.310701+0000","last_became_peered":"2023-10-03T17:34:48.310701+0000","last_unstale":"2023-10-03T17:43:34.128201+0000","last_undegraded":"2023-10-03T17:43:34.128201+0000","last_fullsized":"2023-10-03T17:43:34.128201+0000","mapping_epoch":16,"log_start":"51'51145","ondisk_log_start":"51'51145","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_clean_scrub_stamp":"2023-10-03T17:34:47.281095+0000","objects_scrubbed":1870,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 202s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040768800000000002,"stat_sum":{"num_bytes":5471076352,"num_objects":5138,"num_object_clones":3818,"num_object_copies":10276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5138,"num_whiteouts":0,"num_read":55693,"num_read_kb":205152,"num_write":56534,"num_write_kb":226136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"53'1542","reported_seq":2796,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-10-03T17:43:34.115323+0000","last_change":"2023-10-03T17:40:08.675119+0000","last_active":"2023-10-03T17:43:34.115323+0000","last_peered":"2023-10-03T17:43:34.115323+0000","last_clean":"2023-10-03T17:43:34.115323+0000","last_became_active":"2023-10-03T17:34:42.270625+0000","last_became_peered":"2023-10-03T17:34:42.270625+0000","last_unstale":"2023-10-03T17:43:34.115323+0000","last_undegraded":"2023-10-03T17:43:34.115323+0000","last_fullsized":"2023-10-03T17:43:34.115323+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'841","last_scrub_stamp":"2023-10-03T17:40:08.674910+0000","last_deep_scrub":"46'841","last_deep_scrub_stamp":"2023-10-03T17:40:08.674910+0000","last_clean_scrub_stamp":"2023-10-03T17:40:08.674910+0000","objects_scrubbed":11,"log_size":1542,"log_dups_size":0,"ondisk_log_size":1542,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T21:05:42.790614+0000","scrub_duration":0.039189555000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":706,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":2790,"num_read_kb":2310,"num_write":88,"num_write_kb":88,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2564,"num_omap_keys":60,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"53'56733","reported_seq":105910,"reported_epoch":53,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T17:43:34.109679+0000","last_change":"2023-10-03T17:43:28.494717+0000","last_active":"2023-10-03T17:43:34.109679+0000","last_peered":"2023-10-03T17:43:34.109679+0000","last_clean":"2023-10-03T17:43:34.109679+0000","last_became_active":"2023-10-03T17:34:48.310555+0000","last_became_peered":"2023-10-03T17:34:48.310555+0000","last_unstale":"2023-10-03T17:43:34.109679+0000","last_undegraded":"2023-10-03T17:43:34.109679+0000","last_fullsized":"2023-10-03T17:43:34.109679+0000","mapping_epoch":16,"log_start":"51'46634","ondisk_log_start":"51'46634","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_clean_scrub_stamp":"2023-10-03T17:34:47.281095+0000","objects_scrubbed":93,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 5s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086289699999999995,"stat_sum":{"num_bytes":5149499392,"num_objects":4847,"num_object_clones":3605,"num_object_copies":9694,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4847,"num_whiteouts":0,"num_read":52640,"num_read_kb":193452,"num_write":53057,"num_write_kb":212228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"53'79","reported_seq":1013,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-10-03T17:43:33.691724+0000","last_change":"2023-10-03T17:40:17.641493+0000","last_active":"2023-10-03T17:43:33.691724+0000","last_peered":"2023-10-03T17:43:33.691724+0000","last_clean":"2023-10-03T17:43:33.691724+0000","last_became_active":"2023-10-03T17:34:42.278196+0000","last_became_peered":"2023-10-03T17:34:42.278196+0000","last_unstale":"2023-10-03T17:43:33.691724+0000","last_undegraded":"2023-10-03T17:43:33.691724+0000","last_fullsized":"2023-10-03T17:43:33.691724+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'30","last_scrub_stamp":"2023-10-03T17:40:17.640969+0000","last_deep_scrub":"46'30","last_deep_scrub_stamp":"2023-10-03T17:40:17.640969+0000","last_clean_scrub_stamp":"2023-10-03T17:40:17.640969+0000","objects_scrubbed":6,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:04:02.954896+0000","scrub_duration":0.071054565,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":2065,"num_read_kb":1711,"num_write":47,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":903,"num_omap_keys":30,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"53'59035","reported_seq":109484,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-10-03T17:43:34.116318+0000","last_change":"2023-10-03T17:43:26.193513+0000","last_active":"2023-10-03T17:43:34.116318+0000","last_peered":"2023-10-03T17:43:34.116318+0000","last_clean":"2023-10-03T17:43:34.116318+0000","last_became_active":"2023-10-03T17:34:48.310912+0000","last_became_peered":"2023-10-03T17:34:48.310912+0000","last_unstale":"2023-10-03T17:43:34.116318+0000","last_undegraded":"2023-10-03T17:43:34.116318+0000","last_fullsized":"2023-10-03T17:43:34.116318+0000","mapping_epoch":16,"log_start":"51'48939","ondisk_log_start":"51'48939","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'58196","last_scrub_stamp":"2023-10-03T17:43:26.193075+0000","last_deep_scrub":"53'58196","last_deep_scrub_stamp":"2023-10-03T17:43:26.193075+0000","last_clean_scrub_stamp":"2023-10-03T17:43:26.193075+0000","objects_scrubbed":2326,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":196,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T23:38:17.932978+0000","scrub_duration":195.44739730800001,"objects_trimmed":0,"snaptrim_duration":0.00045661900000000001,"stat_sum":{"num_bytes":5270859776,"num_objects":4945,"num_object_clones":3674,"num_object_copies":9890,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4945,"num_whiteouts":0,"num_read":53829,"num_read_kb":199172,"num_write":54339,"num_write_kb":217356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"53'43","reported_seq":414,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-10-03T17:43:28.573157+0000","last_change":"2023-10-03T17:40:10.708344+0000","last_active":"2023-10-03T17:43:28.573157+0000","last_peered":"2023-10-03T17:43:28.573157+0000","last_clean":"2023-10-03T17:43:28.573157+0000","last_became_active":"2023-10-03T17:34:42.272306+0000","last_became_peered":"2023-10-03T17:34:42.272306+0000","last_unstale":"2023-10-03T17:43:28.573157+0000","last_undegraded":"2023-10-03T17:43:28.573157+0000","last_fullsized":"2023-10-03T17:43:28.573157+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'20","last_scrub_stamp":"2023-10-03T17:40:10.708060+0000","last_deep_scrub":"46'20","last_deep_scrub_stamp":"2023-10-03T17:40:10.708060+0000","last_clean_scrub_stamp":"2023-10-03T17:40:10.708060+0000","objects_scrubbed":7,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:03:09.097252+0000","scrub_duration":0.042279491000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":653,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":723,"num_read_kb":581,"num_write":28,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":301,"num_omap_keys":10,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"53'60210","reported_seq":110857,"reported_epoch":53,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T17:43:34.122041+0000","last_change":"2023-10-03T17:40:29.562114+0000","last_active":"2023-10-03T17:43:34.122041+0000","last_peered":"2023-10-03T17:43:34.122041+0000","last_clean":"2023-10-03T17:43:34.122041+0000","last_became_active":"2023-10-03T17:34:48.310612+0000","last_became_peered":"2023-10-03T17:34:48.310612+0000","last_unstale":"2023-10-03T17:43:34.122041+0000","last_undegraded":"2023-10-03T17:43:34.122041+0000","last_fullsized":"2023-10-03T17:43:34.122041+0000","mapping_epoch":16,"log_start":"51'50151","ondisk_log_start":"51'50151","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_clean_scrub_stamp":"2023-10-03T17:34:47.281095+0000","objects_scrubbed":1904,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 184s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022375120000000001,"stat_sum":{"num_bytes":5338693651,"num_objects":5042,"num_object_clones":3754,"num_object_copies":10084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5042,"num_whiteouts":0,"num_read":54143,"num_read_kb":199716,"num_write":55535,"num_write_kb":222134,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"53'13","reported_seq":89,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-10-03T17:43:01.349428+0000","last_change":"2023-10-03T17:40:24.472734+0000","last_active":"2023-10-03T17:43:01.349428+0000","last_peered":"2023-10-03T17:43:01.349428+0000","last_clean":"2023-10-03T17:43:01.349428+0000","last_became_active":"2023-10-03T17:34:42.270194+0000","last_became_peered":"2023-10-03T17:34:42.270194+0000","last_unstale":"2023-10-03T17:43:01.349428+0000","last_undegraded":"2023-10-03T17:43:01.349428+0000","last_fullsized":"2023-10-03T17:43:01.349428+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'8","last_scrub_stamp":"2023-10-03T17:40:24.472573+0000","last_deep_scrub":"46'8","last_deep_scrub_stamp":"2023-10-03T17:40:24.472573+0000","last_clean_scrub_stamp":"2023-10-03T17:40:24.472573+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T18:28:04.735033+0000","scrub_duration":0.024990637999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":385,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":26,"num_read_kb":8,"num_write":9,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"46'99","reported_seq":194,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-10-03T17:43:00.295007+0000","last_change":"2023-10-03T17:40:10.452341+0000","last_active":"2023-10-03T17:43:00.295007+0000","last_peered":"2023-10-03T17:43:00.295007+0000","last_clean":"2023-10-03T17:43:00.295007+0000","last_became_active":"2023-10-03T17:34:39.502393+0000","last_became_peered":"2023-10-03T17:34:39.502393+0000","last_unstale":"2023-10-03T17:43:00.295007+0000","last_undegraded":"2023-10-03T17:43:00.295007+0000","last_fullsized":"2023-10-03T17:43:00.295007+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'99","last_scrub_stamp":"2023-10-03T17:40:10.449099+0000","last_deep_scrub":"46'99","last_deep_scrub_stamp":"2023-10-03T17:40:10.449099+0000","last_clean_scrub_stamp":"2023-10-03T17:40:10.449099+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T20:18:04.958666+0000","scrub_duration":0.043225012,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":21230129171,"num_objects":19972,"num_object_clones":14851,"num_object_copies":39944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19972,"num_whiteouts":0,"num_read":216305,"num_read_kb":797492,"num_write":219465,"num_write_kb":877854,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1797935104,"data_stored":6113583142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40269,"ondisk_log_size":40269,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4361,"num_objects":123,"num_object_clones":0,"num_object_copies":246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":123,"num_whiteouts":0,"num_read":15230,"num_read_kb":12743,"num_write":443,"num_write_kb":443,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":819200,"data_stored":8722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6405,"ondisk_log_size":6405,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":7460,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738482,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1786488,"kb_used_data":877488,"kb_used_omap":1,"kb_used_meta":908990,"kb_avail":92585352,"statfs":{"total":96636764160,"available":94807400448,"internally_reserved":0,"allocated":898547712,"data_stored":3052183109,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":930806214},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,2,3,3,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 17:42:49 2023","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.44800000000000001,"15min":0.48299999999999998},"min":{"1min":0.17499999999999999,"5min":0.158,"15min":0.14299999999999999},"max":{"1min":4.1539999999999999,"5min":4.4640000000000004,"15min":4.4640000000000004},"last":0.29599999999999999},{"interface":"front","average":{"1min":0.73899999999999999,"5min":0.46200000000000002,"15min":0.501},"min":{"1min":0.17100000000000001,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":4.4180000000000001,"5min":4.4180000000000001,"15min":4.4180000000000001},"last":0.222}]},{"osd":1,"last update":"Tue Oct 3 17:42:49 2023","interfaces":[{"interface":"back","average":{"1min":1.022,"5min":0.502,"15min":0.52200000000000002},"min":{"1min":0.23100000000000001,"5min":0.17599999999999999,"15min":0.16600000000000001},"max":{"1min":5.2359999999999998,"5min":5.2359999999999998,"15min":5.2359999999999998},"last":0.33500000000000002},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.439,"15min":0.51700000000000002},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.192},"max":{"1min":3.9740000000000002,"5min":3.9740000000000002,"15min":3.9740000000000002},"last":0.85099999999999998}]}]},{"osd":1,"up_from":8,"seq":34359738480,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":903080,"kb_used_data":434848,"kb_used_omap":1,"kb_used_meta":468222,"kb_avail":93468760,"statfs":{"total":96636764160,"available":95712010240,"internally_reserved":0,"allocated":445284352,"data_stored":1514642415,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":479459782},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,3,1,5,1,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 17:43:03 2023","interfaces":[{"interface":"back","average":{"1min":0.22600000000000001,"5min":0.31,"15min":0.436},"min":{"1min":0.17399999999999999,"5min":0.161,"15min":0.161},"max":{"1min":0.33700000000000002,"5min":1.8120000000000001,"15min":4.1879999999999997},"last":0.248},{"interface":"front","average":{"1min":0.35699999999999998,"5min":0.33800000000000002,"15min":0.45000000000000001},"min":{"1min":0.20000000000000001,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":1.276,"5min":2.8399999999999999,"15min":2.8399999999999999},"last":0.224}]},{"osd":2,"last update":"Tue Oct 3 17:43:03 2023","interfaces":[{"interface":"back","average":{"1min":0.31,"5min":0.35399999999999998,"15min":0.45600000000000002},"min":{"1min":0.187,"5min":0.17799999999999999,"15min":0.159},"max":{"1min":0.41999999999999998,"5min":3.0390000000000001,"15min":3.0390000000000001},"last":2.6110000000000002},{"interface":"front","average":{"1min":0.28499999999999998,"5min":0.34899999999999998,"15min":0.47299999999999998},"min":{"1min":0.224,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":0.35099999999999998,"5min":3.0699999999999998,"15min":3.0699999999999998},"last":0.28000000000000003}]}]},{"osd":0,"up_from":8,"seq":34359738480,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":926128,"kb_used_data":444008,"kb_used_omap":1,"kb_used_meta":482110,"kb_avail":93445712,"statfs":{"total":96636764160,"available":95688409088,"internally_reserved":0,"allocated":454664192,"data_stored":1540851293,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":493681094},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,2,1,8,9,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 17:42:49 2023","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.47399999999999998,"15min":0.443},"min":{"1min":0.191,"5min":0.154,"15min":0.154},"max":{"1min":2.1349999999999998,"5min":5.7649999999999997,"15min":5.7649999999999997},"last":0.26100000000000001},{"interface":"front","average":{"1min":1.1419999999999999,"5min":0.54300000000000004,"15min":0.49399999999999999},"min":{"1min":0.20499999999999999,"5min":0.19,"15min":0.17699999999999999},"max":{"1min":9.4770000000000003,"5min":9.4770000000000003,"15min":9.4770000000000003},"last":0.28199999999999997}]},{"osd":2,"last update":"Tue Oct 3 17:42:49 2023","interfaces":[{"interface":"back","average":{"1min":1.018,"5min":0.53600000000000003,"15min":0.54300000000000004},"min":{"1min":0.24199999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":9.4149999999999991,"5min":9.4149999999999991,"15min":9.4149999999999991},"last":0.434},{"interface":"front","average":{"1min":0.38600000000000001,"5min":0.42899999999999999,"15min":0.49099999999999999},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":1.5509999999999999,"5min":2.9239999999999999,"15min":2.9239999999999999},"last":0.46800000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":278528,"data_stored":2954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":409600,"data_stored":4361,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":454144000,"data_stored":1540673536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":444792832,"data_stored":1516003347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":898998272,"data_stored":3056906259,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-03T17:43:38.648 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-03T17:34:47.281095+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=34, tm_sec=47, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=40, tm_sec=6, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T17:43:38.649 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-03T17:34:47.281095+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=34, tm_sec=47, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=40, tm_sec=6, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T17:43:38.649 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-03T17:34:47.281095+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=34, tm_sec=47, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=40, tm_sec=6, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T17:43:38.649 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T17:43:58.650 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T17:43:59.133 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-03T17:43:59.134 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-10-03T17:43:59.182 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":332,"stamp":"2023-10-03T17:43:58.079343+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":21280641852,"num_objects":20462,"num_object_clones":15216,"num_object_copies":40924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20462,"num_whiteouts":0,"num_read":239800,"num_read_kb":842222,"num_write":228441,"num_write_kb":913775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":46906,"ondisk_log_size":46906,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3865460,"kb_used_data":1822876,"kb_used_omap":4,"kb_used_meta":2042555,"kb_avail":279250060,"statfs":{"total":289910292480,"available":285952061440,"internally_reserved":0,"allocated":1866625024,"data_stored":6307684497,"data_compressed":7460,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4782,"internal_metadata":2091576658},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,2,7,1,6,9,9,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":23420928,"num_objects":93,"num_object_clones":93,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":93,"num_whiteouts":0,"num_read":3924,"num_read_kb":15344,"num_write":4202,"num_write_kb":16808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002334"},"pg_stats":[{"pgid":"2.7","version":"53'1204","reported_seq":2161,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-10-03T17:43:54.166130+0000","last_change":"2023-10-03T17:40:17.407011+0000","last_active":"2023-10-03T17:43:54.166130+0000","last_peered":"2023-10-03T17:43:54.166130+0000","last_clean":"2023-10-03T17:43:54.166130+0000","last_became_active":"2023-10-03T17:34:42.276624+0000","last_became_peered":"2023-10-03T17:34:42.276624+0000","last_unstale":"2023-10-03T17:43:54.166130+0000","last_undegraded":"2023-10-03T17:43:54.166130+0000","last_fullsized":"2023-10-03T17:43:54.166130+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'673","last_scrub_stamp":"2023-10-03T17:40:17.406813+0000","last_deep_scrub":"46'673","last_deep_scrub_stamp":"2023-10-03T17:40:17.406813+0000","last_clean_scrub_stamp":"2023-10-03T17:40:17.406813+0000","objects_scrubbed":7,"log_size":1204,"log_dups_size":0,"ondisk_log_size":1204,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T21:38:35.056889+0000","scrub_duration":0.013778076,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":537,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":2079,"num_read_kb":1716,"num_write":51,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":903,"num_omap_keys":30,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"53'2021","reported_seq":4172,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-10-03T17:43:54.167062+0000","last_change":"2023-10-03T17:40:13.376519+0000","last_active":"2023-10-03T17:43:54.167062+0000","last_peered":"2023-10-03T17:43:54.167062+0000","last_clean":"2023-10-03T17:43:54.167062+0000","last_became_active":"2023-10-03T17:34:42.271201+0000","last_became_peered":"2023-10-03T17:34:42.271201+0000","last_unstale":"2023-10-03T17:43:54.167062+0000","last_undegraded":"2023-10-03T17:43:54.167062+0000","last_fullsized":"2023-10-03T17:43:54.167062+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'1129","last_scrub_stamp":"2023-10-03T17:40:13.376332+0000","last_deep_scrub":"46'1129","last_deep_scrub_stamp":"2023-10-03T17:40:13.376332+0000","last_clean_scrub_stamp":"2023-10-03T17:40:13.376332+0000","objects_scrubbed":11,"log_size":2021,"log_dups_size":0,"ondisk_log_size":2021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T18:59:17.088585+0000","scrub_duration":0.015998675,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":572,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":4368,"num_read_kb":3777,"num_write":120,"num_write_kb":120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2912,"num_omap_keys":93,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"53'27","reported_seq":110,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-10-03T17:43:01.322917+0000","last_change":"2023-10-03T17:40:25.496010+0000","last_active":"2023-10-03T17:43:01.322917+0000","last_peered":"2023-10-03T17:43:01.322917+0000","last_clean":"2023-10-03T17:43:01.322917+0000","last_became_active":"2023-10-03T17:34:42.276941+0000","last_became_peered":"2023-10-03T17:34:42.276941+0000","last_unstale":"2023-10-03T17:43:01.322917+0000","last_undegraded":"2023-10-03T17:43:01.322917+0000","last_fullsized":"2023-10-03T17:43:01.322917+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'14","last_scrub_stamp":"2023-10-03T17:40:25.495827+0000","last_deep_scrub":"46'14","last_deep_scrub_stamp":"2023-10-03T17:40:25.495827+0000","last_clean_scrub_stamp":"2023-10-03T17:40:25.495827+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:05:44.939663+0000","scrub_duration":0.0133078,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":906,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"53'1620","reported_seq":3205,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-10-03T17:43:54.166199+0000","last_change":"2023-10-03T17:40:26.513902+0000","last_active":"2023-10-03T17:43:54.166199+0000","last_peered":"2023-10-03T17:43:54.166199+0000","last_clean":"2023-10-03T17:43:54.166199+0000","last_became_active":"2023-10-03T17:34:42.276459+0000","last_became_peered":"2023-10-03T17:34:42.276459+0000","last_unstale":"2023-10-03T17:43:54.166199+0000","last_undegraded":"2023-10-03T17:43:54.166199+0000","last_fullsized":"2023-10-03T17:43:54.166199+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'964","last_scrub_stamp":"2023-10-03T17:40:26.513729+0000","last_deep_scrub":"46'964","last_deep_scrub_stamp":"2023-10-03T17:40:26.513729+0000","last_clean_scrub_stamp":"2023-10-03T17:40:26.513729+0000","objects_scrubbed":10,"log_size":1620,"log_dups_size":0,"ondisk_log_size":1620,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:41:33.182751+0000","scrub_duration":0.018190280999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":3140,"num_read_kb":2628,"num_write":80,"num_write_kb":80,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1731,"num_omap_keys":80,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"53'63605","reported_seq":117785,"reported_epoch":53,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T17:43:54.261239+0000","last_change":"2023-10-03T17:40:11.716583+0000","last_active":"2023-10-03T17:43:54.261239+0000","last_peered":"2023-10-03T17:43:54.261239+0000","last_clean":"2023-10-03T17:43:54.261239+0000","last_became_active":"2023-10-03T17:34:48.310701+0000","last_became_peered":"2023-10-03T17:34:48.310701+0000","last_unstale":"2023-10-03T17:43:54.261239+0000","last_undegraded":"2023-10-03T17:43:54.261239+0000","last_fullsized":"2023-10-03T17:43:54.261239+0000","mapping_epoch":16,"log_start":"51'53547","ondisk_log_start":"51'53547","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_clean_scrub_stamp":"2023-10-03T17:34:47.281095+0000","objects_scrubbed":2014,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 222s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040768800000000002,"stat_sum":{"num_bytes":5484314624,"num_objects":5240,"num_object_clones":3920,"num_object_copies":10480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5240,"num_whiteouts":0,"num_read":57810,"num_read_kb":213432,"num_write":58766,"num_write_kb":235064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"53'1574","reported_seq":2844,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-10-03T17:43:54.166915+0000","last_change":"2023-10-03T17:40:08.675119+0000","last_active":"2023-10-03T17:43:54.166915+0000","last_peered":"2023-10-03T17:43:54.166915+0000","last_clean":"2023-10-03T17:43:54.166915+0000","last_became_active":"2023-10-03T17:34:42.270625+0000","last_became_peered":"2023-10-03T17:34:42.270625+0000","last_unstale":"2023-10-03T17:43:54.166915+0000","last_undegraded":"2023-10-03T17:43:54.166915+0000","last_fullsized":"2023-10-03T17:43:54.166915+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'841","last_scrub_stamp":"2023-10-03T17:40:08.674910+0000","last_deep_scrub":"46'841","last_deep_scrub_stamp":"2023-10-03T17:40:08.674910+0000","last_clean_scrub_stamp":"2023-10-03T17:40:08.674910+0000","objects_scrubbed":11,"log_size":1574,"log_dups_size":0,"ondisk_log_size":1574,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T21:05:42.790614+0000","scrub_duration":0.039189555000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":706,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":2790,"num_read_kb":2310,"num_write":88,"num_write_kb":88,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2564,"num_omap_keys":60,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"53'59047","reported_seq":110113,"reported_epoch":53,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T17:43:54.349451+0000","last_change":"2023-10-03T17:43:28.494717+0000","last_active":"2023-10-03T17:43:54.349451+0000","last_peered":"2023-10-03T17:43:54.349451+0000","last_clean":"2023-10-03T17:43:54.349451+0000","last_became_active":"2023-10-03T17:34:48.310555+0000","last_became_peered":"2023-10-03T17:34:48.310555+0000","last_unstale":"2023-10-03T17:43:54.349451+0000","last_undegraded":"2023-10-03T17:43:54.349451+0000","last_fullsized":"2023-10-03T17:43:54.349451+0000","mapping_epoch":16,"log_start":"51'49034","ondisk_log_start":"51'49034","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_clean_scrub_stamp":"2023-10-03T17:34:47.281095+0000","objects_scrubbed":378,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 25s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086289699999999995,"stat_sum":{"num_bytes":5162655744,"num_objects":4933,"num_object_clones":3691,"num_object_copies":9866,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4933,"num_whiteouts":0,"num_read":54591,"num_read_kb":201060,"num_write":55064,"num_write_kb":220256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"53'79","reported_seq":1025,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-10-03T17:43:54.179706+0000","last_change":"2023-10-03T17:40:17.641493+0000","last_active":"2023-10-03T17:43:54.179706+0000","last_peered":"2023-10-03T17:43:54.179706+0000","last_clean":"2023-10-03T17:43:54.179706+0000","last_became_active":"2023-10-03T17:34:42.278196+0000","last_became_peered":"2023-10-03T17:34:42.278196+0000","last_unstale":"2023-10-03T17:43:54.179706+0000","last_undegraded":"2023-10-03T17:43:54.179706+0000","last_fullsized":"2023-10-03T17:43:54.179706+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'30","last_scrub_stamp":"2023-10-03T17:40:17.640969+0000","last_deep_scrub":"46'30","last_deep_scrub_stamp":"2023-10-03T17:40:17.640969+0000","last_clean_scrub_stamp":"2023-10-03T17:40:17.640969+0000","objects_scrubbed":6,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:04:02.954896+0000","scrub_duration":0.071054565,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":2065,"num_read_kb":1711,"num_write":47,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":903,"num_omap_keys":30,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"53'61202","reported_seq":113592,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-10-03T17:43:54.269080+0000","last_change":"2023-10-03T17:43:26.193513+0000","last_active":"2023-10-03T17:43:54.269080+0000","last_peered":"2023-10-03T17:43:54.269080+0000","last_clean":"2023-10-03T17:43:54.269080+0000","last_became_active":"2023-10-03T17:34:48.310912+0000","last_became_peered":"2023-10-03T17:34:48.310912+0000","last_unstale":"2023-10-03T17:43:54.269080+0000","last_undegraded":"2023-10-03T17:43:54.269080+0000","last_fullsized":"2023-10-03T17:43:54.269080+0000","mapping_epoch":16,"log_start":"51'51140","ondisk_log_start":"51'51140","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'58196","last_scrub_stamp":"2023-10-03T17:43:26.193075+0000","last_deep_scrub":"53'58196","last_deep_scrub_stamp":"2023-10-03T17:43:26.193075+0000","last_clean_scrub_stamp":"2023-10-03T17:43:26.193075+0000","objects_scrubbed":2326,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":196,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T23:38:17.932978+0000","scrub_duration":195.44739730800001,"objects_trimmed":0,"snaptrim_duration":0.00045661900000000001,"stat_sum":{"num_bytes":5282832384,"num_objects":5038,"num_object_clones":3767,"num_object_copies":10076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5038,"num_whiteouts":0,"num_read":55866,"num_read_kb":207132,"num_write":56413,"num_write_kb":225652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"53'43","reported_seq":418,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-10-03T17:43:49.046222+0000","last_change":"2023-10-03T17:40:10.708344+0000","last_active":"2023-10-03T17:43:49.046222+0000","last_peered":"2023-10-03T17:43:49.046222+0000","last_clean":"2023-10-03T17:43:49.046222+0000","last_became_active":"2023-10-03T17:34:42.272306+0000","last_became_peered":"2023-10-03T17:34:42.272306+0000","last_unstale":"2023-10-03T17:43:49.046222+0000","last_undegraded":"2023-10-03T17:43:49.046222+0000","last_fullsized":"2023-10-03T17:43:49.046222+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'20","last_scrub_stamp":"2023-10-03T17:40:10.708060+0000","last_deep_scrub":"46'20","last_deep_scrub_stamp":"2023-10-03T17:40:10.708060+0000","last_clean_scrub_stamp":"2023-10-03T17:40:10.708060+0000","objects_scrubbed":7,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:03:09.097252+0000","scrub_duration":0.042279491000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":653,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":723,"num_read_kb":581,"num_write":28,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":301,"num_omap_keys":10,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"53'62546","reported_seq":115183,"reported_epoch":53,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T17:43:54.352434+0000","last_change":"2023-10-03T17:40:29.562114+0000","last_active":"2023-10-03T17:43:54.352434+0000","last_peered":"2023-10-03T17:43:54.352434+0000","last_clean":"2023-10-03T17:43:54.352434+0000","last_became_active":"2023-10-03T17:34:48.310612+0000","last_became_peered":"2023-10-03T17:34:48.310612+0000","last_unstale":"2023-10-03T17:43:54.352434+0000","last_undegraded":"2023-10-03T17:43:54.352434+0000","last_fullsized":"2023-10-03T17:43:54.352434+0000","mapping_epoch":16,"log_start":"51'52453","ondisk_log_start":"51'52453","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_clean_scrub_stamp":"2023-10-03T17:34:47.281095+0000","objects_scrubbed":2128,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 204s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022375120000000001,"stat_sum":{"num_bytes":5350244371,"num_objects":5126,"num_object_clones":3838,"num_object_copies":10252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5126,"num_whiteouts":0,"num_read":56197,"num_read_kb":207764,"num_write":57618,"num_write_kb":230466,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"53'13","reported_seq":89,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-10-03T17:43:01.349428+0000","last_change":"2023-10-03T17:40:24.472734+0000","last_active":"2023-10-03T17:43:01.349428+0000","last_peered":"2023-10-03T17:43:01.349428+0000","last_clean":"2023-10-03T17:43:01.349428+0000","last_became_active":"2023-10-03T17:34:42.270194+0000","last_became_peered":"2023-10-03T17:34:42.270194+0000","last_unstale":"2023-10-03T17:43:01.349428+0000","last_undegraded":"2023-10-03T17:43:01.349428+0000","last_fullsized":"2023-10-03T17:43:01.349428+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'8","last_scrub_stamp":"2023-10-03T17:40:24.472573+0000","last_deep_scrub":"46'8","last_deep_scrub_stamp":"2023-10-03T17:40:24.472573+0000","last_clean_scrub_stamp":"2023-10-03T17:40:24.472573+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T18:28:04.735033+0000","scrub_duration":0.024990637999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":385,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":26,"num_read_kb":8,"num_write":9,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"46'99","reported_seq":194,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-10-03T17:43:00.295007+0000","last_change":"2023-10-03T17:40:10.452341+0000","last_active":"2023-10-03T17:43:00.295007+0000","last_peered":"2023-10-03T17:43:00.295007+0000","last_clean":"2023-10-03T17:43:00.295007+0000","last_became_active":"2023-10-03T17:34:39.502393+0000","last_became_peered":"2023-10-03T17:34:39.502393+0000","last_unstale":"2023-10-03T17:43:00.295007+0000","last_undegraded":"2023-10-03T17:43:00.295007+0000","last_fullsized":"2023-10-03T17:43:00.295007+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'99","last_scrub_stamp":"2023-10-03T17:40:10.449099+0000","last_deep_scrub":"46'99","last_deep_scrub_stamp":"2023-10-03T17:40:10.449099+0000","last_clean_scrub_stamp":"2023-10-03T17:40:10.449099+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T20:18:04.958666+0000","scrub_duration":0.043225012,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":21280047123,"num_objects":20337,"num_object_clones":15216,"num_object_copies":40674,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20337,"num_whiteouts":0,"num_read":224464,"num_read_kb":829388,"num_write":227861,"num_write_kb":911438,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1866575872,"data_stored":6308720678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40226,"ondisk_log_size":40226,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4361,"num_objects":123,"num_object_clones":0,"num_object_copies":246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":123,"num_whiteouts":0,"num_read":15230,"num_read_kb":12743,"num_write":443,"num_write_kb":443,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":819200,"data_stored":8722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6581,"ondisk_log_size":6581,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":7460,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738486,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1890344,"kb_used_data":910560,"kb_used_omap":1,"kb_used_meta":979774,"kb_avail":92481496,"statfs":{"total":96636764160,"available":94701051904,"internally_reserved":0,"allocated":932413440,"data_stored":3152535109,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1003289030},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,2,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 17:43:53 2023","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.42799999999999999,"15min":0.47399999999999998},"min":{"1min":0.16900000000000001,"5min":0.158,"15min":0.14299999999999999},"max":{"1min":1.496,"5min":4.1539999999999999,"15min":4.4640000000000004},"last":0.25700000000000001},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.46500000000000002,"15min":0.48799999999999999},"min":{"1min":0.17799999999999999,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":1.653,"5min":4.4180000000000001,"15min":4.4180000000000001},"last":0.45600000000000002}]},{"osd":1,"last update":"Tue Oct 3 17:43:53 2023","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.48799999999999999,"15min":0.51200000000000001},"min":{"1min":0.214,"5min":0.17599999999999999,"15min":0.16600000000000001},"max":{"1min":2.1520000000000001,"5min":5.2359999999999998,"15min":5.2359999999999998},"last":8.1059999999999999},{"interface":"front","average":{"1min":0.34699999999999998,"5min":0.436,"15min":0.5},"min":{"1min":0.20999999999999999,"5min":0.20200000000000001,"15min":0.192},"max":{"1min":0.85099999999999998,"5min":3.9740000000000002,"15min":3.9740000000000002},"last":0.24299999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738484,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":977864,"kb_used_data":451328,"kb_used_omap":1,"kb_used_meta":526526,"kb_avail":93393976,"statfs":{"total":96636764160,"available":95635431424,"internally_reserved":0,"allocated":462159872,"data_stored":1562635247,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":539163078},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,3,0,2,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 17:43:03 2023","interfaces":[{"interface":"back","average":{"1min":0.22600000000000001,"5min":0.31,"15min":0.436},"min":{"1min":0.17399999999999999,"5min":0.161,"15min":0.161},"max":{"1min":0.33700000000000002,"5min":1.8120000000000001,"15min":4.1879999999999997},"last":0.23300000000000001},{"interface":"front","average":{"1min":0.35699999999999998,"5min":0.33800000000000002,"15min":0.45000000000000001},"min":{"1min":0.20000000000000001,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":1.276,"5min":2.8399999999999999,"15min":2.8399999999999999},"last":0.32600000000000001}]},{"osd":2,"last update":"Tue Oct 3 17:43:03 2023","interfaces":[{"interface":"back","average":{"1min":0.31,"5min":0.35399999999999998,"15min":0.45600000000000002},"min":{"1min":0.187,"5min":0.17799999999999999,"15min":0.159},"max":{"1min":0.41999999999999998,"5min":3.0390000000000001,"15min":3.0390000000000001},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.28499999999999998,"5min":0.34899999999999998,"15min":0.47299999999999998},"min":{"1min":0.224,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":0.35099999999999998,"5min":3.0699999999999998,"15min":3.0699999999999998},"last":0.372}]}]},{"osd":0,"up_from":8,"seq":34359738484,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":997252,"kb_used_data":460988,"kb_used_omap":1,"kb_used_meta":536254,"kb_avail":93374588,"statfs":{"total":96636764160,"available":95615578112,"internally_reserved":0,"allocated":472051712,"data_stored":1592514141,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":549124550},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,3,0,2,3,7,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 17:43:51 2023","interfaces":[{"interface":"back","average":{"1min":0.26000000000000001,"5min":0.46999999999999997,"15min":0.43099999999999999},"min":{"1min":0.17100000000000001,"5min":0.154,"15min":0.154},"max":{"1min":0.38100000000000001,"5min":5.7649999999999997,"15min":5.7649999999999997},"last":0.28799999999999998},{"interface":"front","average":{"1min":0.29499999999999998,"5min":0.51100000000000001,"15min":0.48199999999999998},"min":{"1min":0.191,"5min":0.19,"15min":0.17699999999999999},"max":{"1min":0.38700000000000001,"5min":9.4770000000000003,"15min":9.4770000000000003},"last":0.32200000000000001}]},{"osd":2,"last update":"Tue Oct 3 17:43:51 2023","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.53200000000000003,"15min":0.54400000000000004},"min":{"1min":0.222,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":3.8239999999999998,"5min":9.4149999999999991,"15min":9.4149999999999991},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.42899999999999999,"15min":0.49199999999999999},"min":{"1min":0.27300000000000002,"5min":0.189,"15min":0.189},"max":{"1min":3.8100000000000001,"5min":3.8100000000000001,"15min":3.8100000000000001},"last":0.443}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":278528,"data_stored":2954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":409600,"data_stored":4361,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":471785472,"data_stored":1592250368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":461557760,"data_stored":1562165267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":933232640,"data_stored":3154305043,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-03T17:43:59.184 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-03T17:34:47.281095+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=34, tm_sec=47, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=40, tm_sec=6, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T17:43:59.184 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-03T17:34:47.281095+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=34, tm_sec=47, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=40, tm_sec=6, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T17:43:59.185 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-03T17:34:47.281095+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=34, tm_sec=47, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=40, tm_sec=6, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T17:43:59.185 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T17:44:03.537 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:44:03.533+0000 7f09e4851640 -1 reset not still connected to 0x557d76621040 2023-10-03T17:44:03.537 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:44:03.533+0000 7f09e4851640 -1 reset not still connected to 0x557d7664e750 2023-10-03T17:44:03.539 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:44:03.537+0000 7f09e4851640 -1 reset not still connected to 0x557d76621040 2023-10-03T17:44:03.540 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:44:03.537+0000 7f09e4851640 -1 reset not still connected to 0x557d7664e750 2023-10-03T17:44:17.054 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:44:17.052+0000 7f09e4851640 -1 reset not still connected to 0x557d75f2b2b0 2023-10-03T17:44:17.055 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:44:17.052+0000 7f09e4851640 -1 reset not still connected to 0x557d76621040 2023-10-03T17:44:17.058 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:44:17.060+0000 7f09e4851640 -1 reset not still connected to 0x557d7664ea90 2023-10-03T17:44:17.059 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:44:17.060+0000 7f09e4851640 -1 reset not still connected to 0x557d7664f380 2023-10-03T17:44:17.059 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:44:17.060+0000 7f09e4851640 -1 reset not still connected to 0x557d7664f930 2023-10-03T17:44:17.059 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:44:17.060+0000 7f09e4851640 -1 reset not still connected to 0x557d76681ba0 2023-10-03T17:44:17.059 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:44:17.060+0000 7f09e4851640 -1 reset not still connected to 0x557d766a40d0 2023-10-03T17:44:17.059 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:44:17.060+0000 7f09e4851640 -1 reset not still connected to 0x557d766b56c0 2023-10-03T17:44:17.059 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:44:17.060+0000 7f09e4851640 -1 reset not still connected to 0x557d76eca270 2023-10-03T17:44:17.059 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:44:17.060+0000 7f09e4851640 -1 reset not still connected to 0x557d76f93ba0 2023-10-03T17:44:19.186 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T17:44:19.682 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-03T17:44:19.683 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-10-03T17:44:19.711 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":344,"stamp":"2023-10-03T17:44:18.082668+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":21322258223,"num_objects":23542,"num_object_clones":18276,"num_object_copies":47084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23542,"num_whiteouts":0,"num_read":250575,"num_read_kb":872302,"num_write":235841,"num_write_kb":943135,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":47185,"ondisk_log_size":47185,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":4043480,"kb_used_data":1882880,"kb_used_omap":4,"kb_used_meta":2160571,"kb_avail":279072040,"statfs":{"total":289910292480,"available":285769768960,"internally_reserved":0,"allocated":1928069120,"data_stored":7479436645,"data_compressed":7460,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4782,"internal_metadata":2212425042},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,4,12,11,14,15,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":20242432,"num_objects":2012,"num_object_clones":2012,"num_object_copies":4024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2012,"num_whiteouts":0,"num_read":4812,"num_read_kb":14387,"num_write":3525,"num_write_kb":14100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001964"},"pg_stats":[{"pgid":"2.7","version":"55'1262","reported_seq":2344,"reported_epoch":55,"state":"active+clean","last_fresh":"2023-10-03T17:44:11.643123+0000","last_change":"2023-10-03T17:40:17.407011+0000","last_active":"2023-10-03T17:44:11.643123+0000","last_peered":"2023-10-03T17:44:11.643123+0000","last_clean":"2023-10-03T17:44:11.643123+0000","last_became_active":"2023-10-03T17:34:42.276624+0000","last_became_peered":"2023-10-03T17:34:42.276624+0000","last_unstale":"2023-10-03T17:44:11.643123+0000","last_undegraded":"2023-10-03T17:44:11.643123+0000","last_fullsized":"2023-10-03T17:44:11.643123+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'673","last_scrub_stamp":"2023-10-03T17:40:17.406813+0000","last_deep_scrub":"46'673","last_deep_scrub_stamp":"2023-10-03T17:40:17.406813+0000","last_clean_scrub_stamp":"2023-10-03T17:40:17.406813+0000","objects_scrubbed":7,"log_size":1262,"log_dups_size":0,"ondisk_log_size":1262,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T21:38:35.056889+0000","scrub_duration":0.013778076,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":789,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":2625,"num_read_kb":2157,"num_write":65,"num_write_kb":65,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":903,"num_omap_keys":30,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"55'2115","reported_seq":4599,"reported_epoch":55,"state":"active+clean","last_fresh":"2023-10-03T17:44:11.646306+0000","last_change":"2023-10-03T17:40:13.376519+0000","last_active":"2023-10-03T17:44:11.646306+0000","last_peered":"2023-10-03T17:44:11.646306+0000","last_clean":"2023-10-03T17:44:11.646306+0000","last_became_active":"2023-10-03T17:34:42.271201+0000","last_became_peered":"2023-10-03T17:34:42.271201+0000","last_unstale":"2023-10-03T17:44:11.646306+0000","last_undegraded":"2023-10-03T17:44:11.646306+0000","last_fullsized":"2023-10-03T17:44:11.646306+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'1129","last_scrub_stamp":"2023-10-03T17:40:13.376332+0000","last_deep_scrub":"46'1129","last_deep_scrub_stamp":"2023-10-03T17:40:13.376332+0000","last_clean_scrub_stamp":"2023-10-03T17:40:13.376332+0000","objects_scrubbed":11,"log_size":2115,"log_dups_size":0,"ondisk_log_size":2115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T18:59:17.088585+0000","scrub_duration":0.015998675,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":724,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":5443,"num_read_kb":4676,"num_write":138,"num_write_kb":138,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2912,"num_omap_keys":93,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"55'28","reported_seq":113,"reported_epoch":55,"state":"active+clean","last_fresh":"2023-10-03T17:44:01.292188+0000","last_change":"2023-10-03T17:40:25.496010+0000","last_active":"2023-10-03T17:44:01.292188+0000","last_peered":"2023-10-03T17:44:01.292188+0000","last_clean":"2023-10-03T17:44:01.292188+0000","last_became_active":"2023-10-03T17:34:42.276941+0000","last_became_peered":"2023-10-03T17:34:42.276941+0000","last_unstale":"2023-10-03T17:44:01.292188+0000","last_undegraded":"2023-10-03T17:44:01.292188+0000","last_fullsized":"2023-10-03T17:44:01.292188+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'14","last_scrub_stamp":"2023-10-03T17:40:25.495827+0000","last_deep_scrub":"46'14","last_deep_scrub_stamp":"2023-10-03T17:40:25.495827+0000","last_clean_scrub_stamp":"2023-10-03T17:40:25.495827+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:05:44.939663+0000","scrub_duration":0.0133078,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":957,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":21,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"55'1694","reported_seq":3507,"reported_epoch":55,"state":"active+clean","last_fresh":"2023-10-03T17:44:11.643313+0000","last_change":"2023-10-03T17:40:26.513902+0000","last_active":"2023-10-03T17:44:11.643313+0000","last_peered":"2023-10-03T17:44:11.643313+0000","last_clean":"2023-10-03T17:44:11.643313+0000","last_became_active":"2023-10-03T17:34:42.276459+0000","last_became_peered":"2023-10-03T17:34:42.276459+0000","last_unstale":"2023-10-03T17:44:11.643313+0000","last_undegraded":"2023-10-03T17:44:11.643313+0000","last_fullsized":"2023-10-03T17:44:11.643313+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'964","last_scrub_stamp":"2023-10-03T17:40:26.513729+0000","last_deep_scrub":"46'964","last_deep_scrub_stamp":"2023-10-03T17:40:26.513729+0000","last_clean_scrub_stamp":"2023-10-03T17:40:26.513729+0000","objects_scrubbed":10,"log_size":1694,"log_dups_size":0,"ondisk_log_size":1694,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:41:33.182751+0000","scrub_duration":0.018190280999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":334,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3930,"num_read_kb":3277,"num_write":94,"num_write_kb":94,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1731,"num_omap_keys":80,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"55'66405","reported_seq":121610,"reported_epoch":55,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T17:44:14.376997+0000","last_change":"2023-10-03T17:40:11.716583+0000","last_active":"2023-10-03T17:44:14.376997+0000","last_peered":"2023-10-03T17:44:14.376997+0000","last_clean":"2023-10-03T17:44:14.376997+0000","last_became_active":"2023-10-03T17:34:48.310701+0000","last_became_peered":"2023-10-03T17:34:48.310701+0000","last_unstale":"2023-10-03T17:44:14.376997+0000","last_undegraded":"2023-10-03T17:44:14.376997+0000","last_fullsized":"2023-10-03T17:44:14.376997+0000","mapping_epoch":16,"log_start":"51'56357","ondisk_log_start":"51'56357","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_clean_scrub_stamp":"2023-10-03T17:34:47.281095+0000","objects_scrubbed":2152,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 242s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040768800000000002,"stat_sum":{"num_bytes":5494595584,"num_objects":6027,"num_object_clones":4707,"num_object_copies":12054,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6027,"num_whiteouts":0,"num_read":59605,"num_read_kb":220468,"num_write":60677,"num_write_kb":242708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"55'1660","reported_seq":3098,"reported_epoch":55,"state":"active+clean","last_fresh":"2023-10-03T17:44:13.616839+0000","last_change":"2023-10-03T17:40:08.675119+0000","last_active":"2023-10-03T17:44:13.616839+0000","last_peered":"2023-10-03T17:44:13.616839+0000","last_clean":"2023-10-03T17:44:13.616839+0000","last_became_active":"2023-10-03T17:34:42.270625+0000","last_became_peered":"2023-10-03T17:34:42.270625+0000","last_unstale":"2023-10-03T17:44:13.616839+0000","last_undegraded":"2023-10-03T17:44:13.616839+0000","last_fullsized":"2023-10-03T17:44:13.616839+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'841","last_scrub_stamp":"2023-10-03T17:40:08.674910+0000","last_deep_scrub":"46'841","last_deep_scrub_stamp":"2023-10-03T17:40:08.674910+0000","last_clean_scrub_stamp":"2023-10-03T17:40:08.674910+0000","objects_scrubbed":11,"log_size":1660,"log_dups_size":0,"ondisk_log_size":1660,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T21:05:42.790614+0000","scrub_duration":0.039189555000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":857,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":3520,"num_read_kb":2901,"num_write":103,"num_write_kb":103,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2564,"num_omap_keys":60,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"55'61800","reported_seq":113853,"reported_epoch":55,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T17:44:14.542072+0000","last_change":"2023-10-03T17:43:28.494717+0000","last_active":"2023-10-03T17:44:14.542072+0000","last_peered":"2023-10-03T17:44:14.542072+0000","last_clean":"2023-10-03T17:44:14.542072+0000","last_became_active":"2023-10-03T17:34:48.310555+0000","last_became_peered":"2023-10-03T17:34:48.310555+0000","last_unstale":"2023-10-03T17:44:14.542072+0000","last_undegraded":"2023-10-03T17:44:14.542072+0000","last_fullsized":"2023-10-03T17:44:14.542072+0000","mapping_epoch":16,"log_start":"51'51742","ondisk_log_start":"51'51742","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_clean_scrub_stamp":"2023-10-03T17:34:47.281095+0000","objects_scrubbed":663,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 46s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086289699999999995,"stat_sum":{"num_bytes":5173055488,"num_objects":5688,"num_object_clones":4446,"num_object_copies":11376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5688,"num_whiteouts":0,"num_read":56310,"num_read_kb":207776,"num_write":56843,"num_write_kb":227372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"55'92","reported_seq":1163,"reported_epoch":55,"state":"active+clean","last_fresh":"2023-10-03T17:44:09.526750+0000","last_change":"2023-10-03T17:40:17.641493+0000","last_active":"2023-10-03T17:44:09.526750+0000","last_peered":"2023-10-03T17:44:09.526750+0000","last_clean":"2023-10-03T17:44:09.526750+0000","last_became_active":"2023-10-03T17:34:42.278196+0000","last_became_peered":"2023-10-03T17:34:42.278196+0000","last_unstale":"2023-10-03T17:44:09.526750+0000","last_undegraded":"2023-10-03T17:44:09.526750+0000","last_fullsized":"2023-10-03T17:44:09.526750+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'30","last_scrub_stamp":"2023-10-03T17:40:17.640969+0000","last_deep_scrub":"46'30","last_deep_scrub_stamp":"2023-10-03T17:40:17.640969+0000","last_clean_scrub_stamp":"2023-10-03T17:40:17.640969+0000","objects_scrubbed":6,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:04:02.954896+0000","scrub_duration":0.071054565,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":420,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":2611,"num_read_kb":2152,"num_write":57,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":903,"num_omap_keys":30,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"55'63789","reported_seq":117098,"reported_epoch":55,"state":"active+clean","last_fresh":"2023-10-03T17:44:14.376283+0000","last_change":"2023-10-03T17:43:26.193513+0000","last_active":"2023-10-03T17:44:14.376283+0000","last_peered":"2023-10-03T17:44:14.376283+0000","last_clean":"2023-10-03T17:44:14.376283+0000","last_became_active":"2023-10-03T17:34:48.310912+0000","last_became_peered":"2023-10-03T17:34:48.310912+0000","last_unstale":"2023-10-03T17:44:14.376283+0000","last_undegraded":"2023-10-03T17:44:14.376283+0000","last_fullsized":"2023-10-03T17:44:14.376283+0000","mapping_epoch":16,"log_start":"51'53748","ondisk_log_start":"51'53748","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'58196","last_scrub_stamp":"2023-10-03T17:43:26.193075+0000","last_deep_scrub":"53'58196","last_deep_scrub_stamp":"2023-10-03T17:43:26.193075+0000","last_clean_scrub_stamp":"2023-10-03T17:43:26.193075+0000","objects_scrubbed":2326,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":196,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:41:46.955185+0000","scrub_duration":195.44739730800001,"objects_trimmed":0,"snaptrim_duration":0.00045661900000000001,"stat_sum":{"num_bytes":5292859392,"num_objects":5800,"num_object_clones":4529,"num_object_copies":11600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5800,"num_whiteouts":0,"num_read":57542,"num_read_kb":213616,"num_write":58238,"num_write_kb":232952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"55'49","reported_seq":467,"reported_epoch":55,"state":"active+clean","last_fresh":"2023-10-03T17:44:08.956436+0000","last_change":"2023-10-03T17:40:10.708344+0000","last_active":"2023-10-03T17:44:08.956436+0000","last_peered":"2023-10-03T17:44:08.956436+0000","last_clean":"2023-10-03T17:44:08.956436+0000","last_became_active":"2023-10-03T17:34:42.272306+0000","last_became_peered":"2023-10-03T17:34:42.272306+0000","last_unstale":"2023-10-03T17:44:08.956436+0000","last_undegraded":"2023-10-03T17:44:08.956436+0000","last_fullsized":"2023-10-03T17:44:08.956436+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'20","last_scrub_stamp":"2023-10-03T17:40:10.708060+0000","last_deep_scrub":"46'20","last_deep_scrub_stamp":"2023-10-03T17:40:10.708060+0000","last_clean_scrub_stamp":"2023-10-03T17:40:10.708060+0000","objects_scrubbed":7,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:03:09.097252+0000","scrub_duration":0.042279491000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":754,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":905,"num_read_kb":728,"num_write":33,"num_write_kb":33,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":301,"num_omap_keys":10,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"55'65285","reported_seq":118886,"reported_epoch":55,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T17:44:14.534265+0000","last_change":"2023-10-03T17:40:29.562114+0000","last_active":"2023-10-03T17:44:14.534265+0000","last_peered":"2023-10-03T17:44:14.534265+0000","last_clean":"2023-10-03T17:44:14.534265+0000","last_became_active":"2023-10-03T17:34:48.310612+0000","last_became_peered":"2023-10-03T17:34:48.310612+0000","last_unstale":"2023-10-03T17:44:14.534265+0000","last_undegraded":"2023-10-03T17:44:14.534265+0000","last_fullsized":"2023-10-03T17:44:14.534265+0000","mapping_epoch":16,"log_start":"51'55262","ondisk_log_start":"51'55262","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_clean_scrub_stamp":"2023-10-03T17:34:47.281095+0000","objects_scrubbed":2373,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 224s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022375120000000001,"stat_sum":{"num_bytes":5361152019,"num_objects":5882,"num_object_clones":4594,"num_object_copies":11764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5882,"num_whiteouts":0,"num_read":57913,"num_read_kb":214440,"num_write":59423,"num_write_kb":237686,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"55'16","reported_seq":94,"reported_epoch":55,"state":"active+clean","last_fresh":"2023-10-03T17:44:01.337431+0000","last_change":"2023-10-03T17:40:24.472734+0000","last_active":"2023-10-03T17:44:01.337431+0000","last_peered":"2023-10-03T17:44:01.337431+0000","last_clean":"2023-10-03T17:44:01.337431+0000","last_became_active":"2023-10-03T17:34:42.270194+0000","last_became_peered":"2023-10-03T17:34:42.270194+0000","last_unstale":"2023-10-03T17:44:01.337431+0000","last_undegraded":"2023-10-03T17:44:01.337431+0000","last_fullsized":"2023-10-03T17:44:01.337431+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'8","last_scrub_stamp":"2023-10-03T17:40:24.472573+0000","last_deep_scrub":"46'8","last_deep_scrub_stamp":"2023-10-03T17:40:24.472573+0000","last_clean_scrub_stamp":"2023-10-03T17:40:24.472573+0000","objects_scrubbed":4,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T18:28:04.735033+0000","scrub_duration":0.024990637999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":537,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":26,"num_read_kb":8,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"46'99","reported_seq":196,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-03T17:44:00.289890+0000","last_change":"2023-10-03T17:40:10.452341+0000","last_active":"2023-10-03T17:44:00.289890+0000","last_peered":"2023-10-03T17:44:00.289890+0000","last_clean":"2023-10-03T17:44:00.289890+0000","last_became_active":"2023-10-03T17:34:39.502393+0000","last_became_peered":"2023-10-03T17:34:39.502393+0000","last_unstale":"2023-10-03T17:44:00.289890+0000","last_undegraded":"2023-10-03T17:44:00.289890+0000","last_fullsized":"2023-10-03T17:44:00.289890+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'99","last_scrub_stamp":"2023-10-03T17:40:10.449099+0000","last_deep_scrub":"46'99","last_deep_scrub_stamp":"2023-10-03T17:40:10.449099+0000","last_clean_scrub_stamp":"2023-10-03T17:40:10.449099+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T20:18:04.958666+0000","scrub_duration":0.043225012,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":21321662483,"num_objects":23397,"num_object_clones":18276,"num_object_copies":46794,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23397,"num_whiteouts":0,"num_read":231370,"num_read_kb":856300,"num_write":235181,"num_write_kb":940718,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1926455296,"data_stored":7496253478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40170,"ondisk_log_size":40170,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":5372,"num_objects":143,"num_object_clones":0,"num_object_copies":286,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":143,"num_whiteouts":0,"num_read":19099,"num_read_kb":15911,"num_write":523,"num_write_kb":523,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":983040,"data_stored":10744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6916,"ondisk_log_size":6916,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":7460,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738490,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1966668,"kb_used_data":941060,"kb_used_omap":1,"kb_used_meta":1025598,"kb_avail":92405172,"statfs":{"total":96636764160,"available":94622896128,"internally_reserved":0,"allocated":963645440,"data_stored":3744546415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1050212806},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,5,2,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 17:43:53 2023","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.42799999999999999,"15min":0.47399999999999998},"min":{"1min":0.16900000000000001,"5min":0.158,"15min":0.14299999999999999},"max":{"1min":1.496,"5min":4.1539999999999999,"15min":4.4640000000000004},"last":0.19},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.46500000000000002,"15min":0.48799999999999999},"min":{"1min":0.17799999999999999,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":1.653,"5min":4.4180000000000001,"15min":4.4180000000000001},"last":0.23300000000000001}]},{"osd":1,"last update":"Tue Oct 3 17:43:53 2023","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.48799999999999999,"15min":0.51200000000000001},"min":{"1min":0.214,"5min":0.17599999999999999,"15min":0.16600000000000001},"max":{"1min":2.1520000000000001,"5min":5.2359999999999998,"15min":5.2359999999999998},"last":0.21299999999999999},{"interface":"front","average":{"1min":0.34699999999999998,"5min":0.436,"15min":0.5},"min":{"1min":0.20999999999999999,"5min":0.20200000000000001,"15min":0.192},"max":{"1min":0.85099999999999998,"5min":3.9740000000000002,"15min":3.9740000000000002},"last":0.372}]}]},{"osd":1,"up_from":8,"seq":34359738488,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1046492,"kb_used_data":465812,"kb_used_omap":1,"kb_used_meta":580670,"kb_avail":93325348,"statfs":{"total":96636764160,"available":95565156352,"internally_reserved":0,"allocated":476991488,"data_stored":1853492956,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":594606534},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,3,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 17:44:07 2023","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.33200000000000002,"15min":0.42599999999999999},"min":{"1min":0.20899999999999999,"5min":0.17100000000000001,"15min":0.161},"max":{"1min":1.341,"5min":1.8120000000000001,"15min":4.1879999999999997},"last":0.22900000000000001},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.433,"15min":0.46400000000000002},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":4.5759999999999996,"5min":4.5759999999999996,"15min":4.5759999999999996},"last":0.254}]},{"osd":2,"last update":"Tue Oct 3 17:44:07 2023","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.39200000000000002,"15min":0.45200000000000001},"min":{"1min":0.20999999999999999,"5min":0.17799999999999999,"15min":0.159},"max":{"1min":2.6110000000000002,"5min":3.0390000000000001,"15min":3.0390000000000001},"last":1.3100000000000001},{"interface":"front","average":{"1min":0.34499999999999997,"5min":0.35799999999999998,"15min":0.45800000000000002},"min":{"1min":0.251,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":0.60099999999999998,"5min":3.0699999999999998,"15min":3.0699999999999998},"last":1.337}]}]},{"osd":0,"up_from":8,"seq":34359738488,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1030320,"kb_used_data":476008,"kb_used_omap":1,"kb_used_meta":554302,"kb_avail":93341520,"statfs":{"total":96636764160,"available":95581716480,"internally_reserved":0,"allocated":487432192,"data_stored":1881397274,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":567605702},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,1,6,2,11,7,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 17:43:51 2023","interfaces":[{"interface":"back","average":{"1min":0.26000000000000001,"5min":0.46999999999999997,"15min":0.43099999999999999},"min":{"1min":0.17100000000000001,"5min":0.154,"15min":0.154},"max":{"1min":0.38100000000000001,"5min":5.7649999999999997,"15min":5.7649999999999997},"last":0.25600000000000001},{"interface":"front","average":{"1min":0.29499999999999998,"5min":0.51100000000000001,"15min":0.48199999999999998},"min":{"1min":0.191,"5min":0.19,"15min":0.17699999999999999},"max":{"1min":0.38700000000000001,"5min":9.4770000000000003,"15min":9.4770000000000003},"last":0.191}]},{"osd":2,"last update":"Tue Oct 3 17:43:51 2023","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.53200000000000003,"15min":0.54400000000000004},"min":{"1min":0.222,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":3.8239999999999998,"5min":9.4149999999999991,"15min":9.4149999999999991},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.42899999999999999,"15min":0.49199999999999999},"min":{"1min":0.27300000000000002,"5min":0.189,"15min":0.189},"max":{"1min":3.8100000000000001,"5min":3.8100000000000001,"15min":3.8100000000000001},"last":0.40999999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":335872,"data_stored":3661,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":5372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":155648,"data_stored":1711,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":487084032,"data_stored":1892458496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":476221440,"data_stored":1857261587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":963149824,"data_stored":3746533395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-03T17:44:19.713 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-03T17:34:47.281095+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=34, tm_sec=47, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=40, tm_sec=6, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T17:44:19.713 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-03T17:34:47.281095+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=34, tm_sec=47, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=40, tm_sec=6, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T17:44:19.714 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-03T17:34:47.281095+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=34, tm_sec=47, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=40, tm_sec=6, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T17:44:19.714 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T17:44:39.715 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T17:44:40.285 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-03T17:44:40.419 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-10-03T17:44:40.419 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":355,"stamp":"2023-10-03T17:44:40.089454+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":21380052783,"num_objects":25266,"num_object_clones":20000,"num_object_copies":50532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25266,"num_whiteouts":0,"num_read":260253,"num_read_kb":910338,"num_write":245351,"num_write_kb":981175,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":47497,"ondisk_log_size":47497,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":4259620,"kb_used_data":1958796,"kb_used_omap":4,"kb_used_meta":2300795,"kb_avail":278855900,"statfs":{"total":289910292480,"available":285548441600,"internally_reserved":0,"allocated":2005807104,"data_stored":8207660389,"data_compressed":7460,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4782,"internal_metadata":2356014418},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,9,8,15,1,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":36552704,"num_objects":722,"num_object_clones":722,"num_object_copies":1444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":722,"num_whiteouts":0,"num_read":5949,"num_read_kb":23420,"num_write":5879,"num_write_kb":23516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005219"},"pg_stats":[{"pgid":"2.7","version":"55'1307","reported_seq":2404,"reported_epoch":55,"state":"active+clean","last_fresh":"2023-10-03T17:44:39.771528+0000","last_change":"2023-10-03T17:40:17.407011+0000","last_active":"2023-10-03T17:44:39.771528+0000","last_peered":"2023-10-03T17:44:39.771528+0000","last_clean":"2023-10-03T17:44:39.771528+0000","last_became_active":"2023-10-03T17:34:42.276624+0000","last_became_peered":"2023-10-03T17:34:42.276624+0000","last_unstale":"2023-10-03T17:44:39.771528+0000","last_undegraded":"2023-10-03T17:44:39.771528+0000","last_fullsized":"2023-10-03T17:44:39.771528+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'673","last_scrub_stamp":"2023-10-03T17:40:17.406813+0000","last_deep_scrub":"46'673","last_deep_scrub_stamp":"2023-10-03T17:40:17.406813+0000","last_clean_scrub_stamp":"2023-10-03T17:40:17.406813+0000","objects_scrubbed":7,"log_size":1307,"log_dups_size":0,"ondisk_log_size":1307,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T21:38:35.056889+0000","scrub_duration":0.013778076,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":789,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":2625,"num_read_kb":2157,"num_write":65,"num_write_kb":65,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":903,"num_omap_keys":30,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"55'2190","reported_seq":4699,"reported_epoch":55,"state":"active+clean","last_fresh":"2023-10-03T17:44:39.773747+0000","last_change":"2023-10-03T17:40:13.376519+0000","last_active":"2023-10-03T17:44:39.773747+0000","last_peered":"2023-10-03T17:44:39.773747+0000","last_clean":"2023-10-03T17:44:39.773747+0000","last_became_active":"2023-10-03T17:34:42.271201+0000","last_became_peered":"2023-10-03T17:34:42.271201+0000","last_unstale":"2023-10-03T17:44:39.773747+0000","last_undegraded":"2023-10-03T17:44:39.773747+0000","last_fullsized":"2023-10-03T17:44:39.773747+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'1129","last_scrub_stamp":"2023-10-03T17:40:13.376332+0000","last_deep_scrub":"46'1129","last_deep_scrub_stamp":"2023-10-03T17:40:13.376332+0000","last_clean_scrub_stamp":"2023-10-03T17:40:13.376332+0000","objects_scrubbed":11,"log_size":2190,"log_dups_size":0,"ondisk_log_size":2190,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T18:59:17.088585+0000","scrub_duration":0.015998675,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":724,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":5443,"num_read_kb":4676,"num_write":138,"num_write_kb":138,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2912,"num_omap_keys":93,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"55'28","reported_seq":113,"reported_epoch":55,"state":"active+clean","last_fresh":"2023-10-03T17:44:01.292188+0000","last_change":"2023-10-03T17:40:25.496010+0000","last_active":"2023-10-03T17:44:01.292188+0000","last_peered":"2023-10-03T17:44:01.292188+0000","last_clean":"2023-10-03T17:44:01.292188+0000","last_became_active":"2023-10-03T17:34:42.276941+0000","last_became_peered":"2023-10-03T17:34:42.276941+0000","last_unstale":"2023-10-03T17:44:01.292188+0000","last_undegraded":"2023-10-03T17:44:01.292188+0000","last_fullsized":"2023-10-03T17:44:01.292188+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'14","last_scrub_stamp":"2023-10-03T17:40:25.495827+0000","last_deep_scrub":"46'14","last_deep_scrub_stamp":"2023-10-03T17:40:25.495827+0000","last_clean_scrub_stamp":"2023-10-03T17:40:25.495827+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:05:44.939663+0000","scrub_duration":0.0133078,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":957,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":21,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"55'1754","reported_seq":3587,"reported_epoch":55,"state":"active+clean","last_fresh":"2023-10-03T17:44:39.773450+0000","last_change":"2023-10-03T17:40:26.513902+0000","last_active":"2023-10-03T17:44:39.773450+0000","last_peered":"2023-10-03T17:44:39.773450+0000","last_clean":"2023-10-03T17:44:39.773450+0000","last_became_active":"2023-10-03T17:34:42.276459+0000","last_became_peered":"2023-10-03T17:34:42.276459+0000","last_unstale":"2023-10-03T17:44:39.773450+0000","last_undegraded":"2023-10-03T17:44:39.773450+0000","last_fullsized":"2023-10-03T17:44:39.773450+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'964","last_scrub_stamp":"2023-10-03T17:40:26.513729+0000","last_deep_scrub":"46'964","last_deep_scrub_stamp":"2023-10-03T17:40:26.513729+0000","last_clean_scrub_stamp":"2023-10-03T17:40:26.513729+0000","objects_scrubbed":10,"log_size":1754,"log_dups_size":0,"ondisk_log_size":1754,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:41:33.182751+0000","scrub_duration":0.018190280999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":334,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3930,"num_read_kb":3277,"num_write":94,"num_write_kb":94,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1731,"num_omap_keys":80,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"55'69513","reported_seq":126719,"reported_epoch":55,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T17:44:39.490578+0000","last_change":"2023-10-03T17:40:11.716583+0000","last_active":"2023-10-03T17:44:39.490578+0000","last_peered":"2023-10-03T17:44:39.490578+0000","last_clean":"2023-10-03T17:44:39.490578+0000","last_became_active":"2023-10-03T17:34:48.310701+0000","last_became_peered":"2023-10-03T17:34:48.310701+0000","last_unstale":"2023-10-03T17:44:39.490578+0000","last_undegraded":"2023-10-03T17:44:39.490578+0000","last_fullsized":"2023-10-03T17:44:39.490578+0000","mapping_epoch":16,"log_start":"53'59465","ondisk_log_start":"53'59465","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_clean_scrub_stamp":"2023-10-03T17:34:47.281095+0000","objects_scrubbed":2417,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 267s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040768800000000002,"stat_sum":{"num_bytes":5509439488,"num_objects":6470,"num_object_clones":5150,"num_object_copies":12940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6470,"num_whiteouts":0,"num_read":62031,"num_read_kb":230016,"num_write":63147,"num_write_kb":252588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"55'1720","reported_seq":3178,"reported_epoch":55,"state":"active+clean","last_fresh":"2023-10-03T17:44:37.268746+0000","last_change":"2023-10-03T17:40:08.675119+0000","last_active":"2023-10-03T17:44:37.268746+0000","last_peered":"2023-10-03T17:44:37.268746+0000","last_clean":"2023-10-03T17:44:37.268746+0000","last_became_active":"2023-10-03T17:34:42.270625+0000","last_became_peered":"2023-10-03T17:34:42.270625+0000","last_unstale":"2023-10-03T17:44:37.268746+0000","last_undegraded":"2023-10-03T17:44:37.268746+0000","last_fullsized":"2023-10-03T17:44:37.268746+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'841","last_scrub_stamp":"2023-10-03T17:40:08.674910+0000","last_deep_scrub":"46'841","last_deep_scrub_stamp":"2023-10-03T17:40:08.674910+0000","last_clean_scrub_stamp":"2023-10-03T17:40:08.674910+0000","objects_scrubbed":11,"log_size":1720,"log_dups_size":0,"ondisk_log_size":1720,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T21:05:42.790614+0000","scrub_duration":0.039189555000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":857,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":3520,"num_read_kb":2901,"num_write":103,"num_write_kb":103,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2564,"num_omap_keys":60,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"55'64832","reported_seq":118917,"reported_epoch":55,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T17:44:39.779563+0000","last_change":"2023-10-03T17:43:28.494717+0000","last_active":"2023-10-03T17:44:39.779563+0000","last_peered":"2023-10-03T17:44:39.779563+0000","last_clean":"2023-10-03T17:44:39.779563+0000","last_became_active":"2023-10-03T17:34:48.310555+0000","last_became_peered":"2023-10-03T17:34:48.310555+0000","last_unstale":"2023-10-03T17:44:39.779563+0000","last_undegraded":"2023-10-03T17:44:39.779563+0000","last_fullsized":"2023-10-03T17:44:39.779563+0000","mapping_epoch":16,"log_start":"53'54748","ondisk_log_start":"53'54748","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_clean_scrub_stamp":"2023-10-03T17:34:47.281095+0000","objects_scrubbed":1014,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 71s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086289699999999995,"stat_sum":{"num_bytes":5188235264,"num_objects":6115,"num_object_clones":4873,"num_object_copies":12230,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6115,"num_whiteouts":0,"num_read":58747,"num_read_kb":217324,"num_write":59171,"num_write_kb":236684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"55'92","reported_seq":1178,"reported_epoch":55,"state":"active+clean","last_fresh":"2023-10-03T17:44:35.167513+0000","last_change":"2023-10-03T17:40:17.641493+0000","last_active":"2023-10-03T17:44:35.167513+0000","last_peered":"2023-10-03T17:44:35.167513+0000","last_clean":"2023-10-03T17:44:35.167513+0000","last_became_active":"2023-10-03T17:34:42.278196+0000","last_became_peered":"2023-10-03T17:34:42.278196+0000","last_unstale":"2023-10-03T17:44:35.167513+0000","last_undegraded":"2023-10-03T17:44:35.167513+0000","last_fullsized":"2023-10-03T17:44:35.167513+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'30","last_scrub_stamp":"2023-10-03T17:40:17.640969+0000","last_deep_scrub":"46'30","last_deep_scrub_stamp":"2023-10-03T17:40:17.640969+0000","last_clean_scrub_stamp":"2023-10-03T17:40:17.640969+0000","objects_scrubbed":6,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:04:02.954896+0000","scrub_duration":0.071054565,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":420,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":2611,"num_read_kb":2152,"num_write":57,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":903,"num_omap_keys":30,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"55'66659","reported_seq":121926,"reported_epoch":55,"state":"active+clean","last_fresh":"2023-10-03T17:44:39.516958+0000","last_change":"2023-10-03T17:43:26.193513+0000","last_active":"2023-10-03T17:44:39.516958+0000","last_peered":"2023-10-03T17:44:39.516958+0000","last_clean":"2023-10-03T17:44:39.516958+0000","last_became_active":"2023-10-03T17:34:48.310912+0000","last_became_peered":"2023-10-03T17:34:48.310912+0000","last_unstale":"2023-10-03T17:44:39.516958+0000","last_undegraded":"2023-10-03T17:44:39.516958+0000","last_fullsized":"2023-10-03T17:44:39.516958+0000","mapping_epoch":16,"log_start":"53'56650","ondisk_log_start":"53'56650","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'58196","last_scrub_stamp":"2023-10-03T17:43:26.193075+0000","last_deep_scrub":"53'58196","last_deep_scrub_stamp":"2023-10-03T17:43:26.193075+0000","last_clean_scrub_stamp":"2023-10-03T17:43:26.193075+0000","objects_scrubbed":2326,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":196,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:41:46.955185+0000","scrub_duration":195.44739730800001,"objects_trimmed":0,"snaptrim_duration":0.00045661900000000001,"stat_sum":{"num_bytes":5306884096,"num_objects":6233,"num_object_clones":4962,"num_object_copies":12466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6233,"num_whiteouts":0,"num_read":59932,"num_read_kb":223028,"num_write":60675,"num_write_kb":242700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"55'49","reported_seq":472,"reported_epoch":55,"state":"active+clean","last_fresh":"2023-10-03T17:44:30.010156+0000","last_change":"2023-10-03T17:40:10.708344+0000","last_active":"2023-10-03T17:44:30.010156+0000","last_peered":"2023-10-03T17:44:30.010156+0000","last_clean":"2023-10-03T17:44:30.010156+0000","last_became_active":"2023-10-03T17:34:42.272306+0000","last_became_peered":"2023-10-03T17:34:42.272306+0000","last_unstale":"2023-10-03T17:44:30.010156+0000","last_undegraded":"2023-10-03T17:44:30.010156+0000","last_fullsized":"2023-10-03T17:44:30.010156+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'20","last_scrub_stamp":"2023-10-03T17:40:10.708060+0000","last_deep_scrub":"46'20","last_deep_scrub_stamp":"2023-10-03T17:40:10.708060+0000","last_clean_scrub_stamp":"2023-10-03T17:40:10.708060+0000","objects_scrubbed":7,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:03:09.097252+0000","scrub_duration":0.042279491000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":754,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":905,"num_read_kb":728,"num_write":33,"num_write_kb":33,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":301,"num_omap_keys":10,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"55'68268","reported_seq":123913,"reported_epoch":55,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T17:44:39.771853+0000","last_change":"2023-10-03T17:40:29.562114+0000","last_active":"2023-10-03T17:44:39.771853+0000","last_peered":"2023-10-03T17:44:39.771853+0000","last_clean":"2023-10-03T17:44:39.771853+0000","last_became_active":"2023-10-03T17:34:48.310612+0000","last_became_peered":"2023-10-03T17:34:48.310612+0000","last_unstale":"2023-10-03T17:44:39.771853+0000","last_undegraded":"2023-10-03T17:44:39.771853+0000","last_fullsized":"2023-10-03T17:44:39.771853+0000","mapping_epoch":16,"log_start":"53'58167","ondisk_log_start":"53'58167","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_clean_scrub_stamp":"2023-10-03T17:34:47.281095+0000","objects_scrubbed":2745,"log_size":10101,"log_dups_size":0,"ondisk_log_size":10101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 250s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022375120000000001,"stat_sum":{"num_bytes":5374898195,"num_objects":6303,"num_object_clones":5015,"num_object_copies":12606,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6303,"num_whiteouts":0,"num_read":60338,"num_read_kb":223968,"num_write":61698,"num_write_kb":246786,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"55'16","reported_seq":94,"reported_epoch":55,"state":"active+clean","last_fresh":"2023-10-03T17:44:01.337431+0000","last_change":"2023-10-03T17:40:24.472734+0000","last_active":"2023-10-03T17:44:01.337431+0000","last_peered":"2023-10-03T17:44:01.337431+0000","last_clean":"2023-10-03T17:44:01.337431+0000","last_became_active":"2023-10-03T17:34:42.270194+0000","last_became_peered":"2023-10-03T17:34:42.270194+0000","last_unstale":"2023-10-03T17:44:01.337431+0000","last_undegraded":"2023-10-03T17:44:01.337431+0000","last_fullsized":"2023-10-03T17:44:01.337431+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'8","last_scrub_stamp":"2023-10-03T17:40:24.472573+0000","last_deep_scrub":"46'8","last_deep_scrub_stamp":"2023-10-03T17:40:24.472573+0000","last_clean_scrub_stamp":"2023-10-03T17:40:24.472573+0000","objects_scrubbed":4,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T18:28:04.735033+0000","scrub_duration":0.024990637999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":537,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":26,"num_read_kb":8,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"46'99","reported_seq":196,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-10-03T17:44:00.289890+0000","last_change":"2023-10-03T17:40:10.452341+0000","last_active":"2023-10-03T17:44:00.289890+0000","last_peered":"2023-10-03T17:44:00.289890+0000","last_clean":"2023-10-03T17:44:00.289890+0000","last_became_active":"2023-10-03T17:34:39.502393+0000","last_became_peered":"2023-10-03T17:34:39.502393+0000","last_unstale":"2023-10-03T17:44:00.289890+0000","last_undegraded":"2023-10-03T17:44:00.289890+0000","last_fullsized":"2023-10-03T17:44:00.289890+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'99","last_scrub_stamp":"2023-10-03T17:40:10.449099+0000","last_deep_scrub":"46'99","last_deep_scrub_stamp":"2023-10-03T17:40:10.449099+0000","last_clean_scrub_stamp":"2023-10-03T17:40:10.449099+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T20:18:04.958666+0000","scrub_duration":0.043225012,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":21379457043,"num_objects":25121,"num_object_clones":20000,"num_object_copies":50242,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25121,"num_whiteouts":0,"num_read":241048,"num_read_kb":894336,"num_write":244691,"num_write_kb":978758,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2004238336,"data_stored":8209682470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40242,"ondisk_log_size":40242,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":5372,"num_objects":143,"num_object_clones":0,"num_object_copies":286,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":143,"num_whiteouts":0,"num_read":19099,"num_read_kb":15911,"num_write":523,"num_write_kb":523,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":983040,"data_stored":10744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7156,"ondisk_log_size":7156,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":7460,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738495,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2117080,"kb_used_data":979024,"kb_used_omap":1,"kb_used_meta":1138046,"kb_avail":92254760,"statfs":{"total":96636764160,"available":94468874240,"internally_reserved":0,"allocated":1002520576,"data_stored":4104236655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1165359558},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 17:43:53 2023","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.42799999999999999,"15min":0.47399999999999998},"min":{"1min":0.16900000000000001,"5min":0.158,"15min":0.14299999999999999},"max":{"1min":1.496,"5min":4.1539999999999999,"15min":4.4640000000000004},"last":0.189},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.46500000000000002,"15min":0.48799999999999999},"min":{"1min":0.17799999999999999,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":1.653,"5min":4.4180000000000001,"15min":4.4180000000000001},"last":0.20999999999999999}]},{"osd":1,"last update":"Tue Oct 3 17:43:53 2023","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.48799999999999999,"15min":0.51200000000000001},"min":{"1min":0.214,"5min":0.17599999999999999,"15min":0.16600000000000001},"max":{"1min":2.1520000000000001,"5min":5.2359999999999998,"15min":5.2359999999999998},"last":0.23200000000000001},{"interface":"front","average":{"1min":0.34699999999999998,"5min":0.436,"15min":0.5},"min":{"1min":0.20999999999999999,"5min":0.20200000000000001,"15min":0.192},"max":{"1min":0.85099999999999998,"5min":3.9740000000000002,"15min":3.9740000000000002},"last":0.17699999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738493,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1070296,"kb_used_data":484048,"kb_used_omap":1,"kb_used_meta":586238,"kb_avail":93301544,"statfs":{"total":96636764160,"available":95540781056,"internally_reserved":0,"allocated":495665152,"data_stored":2030091996,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":600308166},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,7,3,5,1,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 17:44:07 2023","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.33200000000000002,"15min":0.42599999999999999},"min":{"1min":0.20899999999999999,"5min":0.17100000000000001,"15min":0.161},"max":{"1min":1.341,"5min":1.8120000000000001,"15min":4.1879999999999997},"last":0.17199999999999999},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.433,"15min":0.46400000000000002},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":4.5759999999999996,"5min":4.5759999999999996,"15min":4.5759999999999996},"last":0.20599999999999999}]},{"osd":2,"last update":"Tue Oct 3 17:44:07 2023","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.39200000000000002,"15min":0.45200000000000001},"min":{"1min":0.20999999999999999,"5min":0.17799999999999999,"15min":0.159},"max":{"1min":2.6110000000000002,"5min":3.0390000000000001,"15min":3.0390000000000001},"last":0.29199999999999998},{"interface":"front","average":{"1min":0.34499999999999997,"5min":0.35799999999999998,"15min":0.45800000000000002},"min":{"1min":0.251,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":0.60099999999999998,"5min":3.0699999999999998,"15min":3.0699999999999998},"last":0.30599999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738493,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1072244,"kb_used_data":495724,"kb_used_omap":1,"kb_used_meta":576510,"kb_avail":93299596,"statfs":{"total":96636764160,"available":95538786304,"internally_reserved":0,"allocated":507621376,"data_stored":2073331738,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":590346694},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,3,8],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 17:43:51 2023","interfaces":[{"interface":"back","average":{"1min":0.26000000000000001,"5min":0.46999999999999997,"15min":0.43099999999999999},"min":{"1min":0.17100000000000001,"5min":0.154,"15min":0.154},"max":{"1min":0.38100000000000001,"5min":5.7649999999999997,"15min":5.7649999999999997},"last":0.185},{"interface":"front","average":{"1min":0.29499999999999998,"5min":0.51100000000000001,"15min":0.48199999999999998},"min":{"1min":0.191,"5min":0.19,"15min":0.17699999999999999},"max":{"1min":0.38700000000000001,"5min":9.4770000000000003,"15min":9.4770000000000003},"last":0.247}]},{"osd":2,"last update":"Tue Oct 3 17:43:51 2023","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.53200000000000003,"15min":0.54400000000000004},"min":{"1min":0.222,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":3.8239999999999998,"5min":9.4149999999999991,"15min":9.4149999999999991},"last":0.32300000000000001},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.42899999999999999,"15min":0.49199999999999999},"min":{"1min":0.27300000000000002,"5min":0.189,"15min":0.189},"max":{"1min":3.8100000000000001,"5min":3.8100000000000001,"15min":3.8100000000000001},"last":1.9359999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":335872,"data_stored":3661,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":5372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":155648,"data_stored":1711,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":507174912,"data_stored":2074120192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":495087616,"data_stored":2031243283,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1001975808,"data_stored":4104318995,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-03T17:44:40.433 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-03T17:34:47.281095+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=34, tm_sec=47, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=40, tm_sec=6, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T17:44:40.434 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-03T17:34:47.281095+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=34, tm_sec=47, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=40, tm_sec=6, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T17:44:40.434 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-03T17:34:47.281095+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=34, tm_sec=47, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=40, tm_sec=6, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T17:44:40.434 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T17:45:00.435 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T17:45:00.922 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-03T17:45:00.923 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-10-03T17:45:00.956 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":366,"stamp":"2023-10-03T17:45:00.318875+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":21428242223,"num_objects":25566,"num_object_clones":20300,"num_object_copies":51132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25566,"num_whiteouts":0,"num_read":268126,"num_read_kb":940445,"num_write":253381,"num_write_kb":1013295,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":47674,"ondisk_log_size":47674,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":4391004,"kb_used_data":2022212,"kb_used_omap":4,"kb_used_meta":2368763,"kb_avail":278724516,"statfs":{"total":289910292480,"available":285413904384,"internally_reserved":0,"allocated":2070745088,"data_stored":8386061669,"data_compressed":7460,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4782,"internal_metadata":2425613650},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,5,8,6,6,7,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":24387584,"num_objects":112,"num_object_clones":112,"num_object_copies":224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":112,"num_whiteouts":0,"num_read":4087,"num_read_kb":15191,"num_write":4107,"num_write_kb":16428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.227716"},"pg_stats":[{"pgid":"2.7","version":"55'1348","reported_seq":2469,"reported_epoch":55,"state":"active+clean","last_fresh":"2023-10-03T17:45:00.045646+0000","last_change":"2023-10-03T17:40:17.407011+0000","last_active":"2023-10-03T17:45:00.045646+0000","last_peered":"2023-10-03T17:45:00.045646+0000","last_clean":"2023-10-03T17:45:00.045646+0000","last_became_active":"2023-10-03T17:34:42.276624+0000","last_became_peered":"2023-10-03T17:34:42.276624+0000","last_unstale":"2023-10-03T17:45:00.045646+0000","last_undegraded":"2023-10-03T17:45:00.045646+0000","last_fullsized":"2023-10-03T17:45:00.045646+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'673","last_scrub_stamp":"2023-10-03T17:40:17.406813+0000","last_deep_scrub":"46'673","last_deep_scrub_stamp":"2023-10-03T17:40:17.406813+0000","last_clean_scrub_stamp":"2023-10-03T17:40:17.406813+0000","objects_scrubbed":7,"log_size":1348,"log_dups_size":0,"ondisk_log_size":1348,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T21:38:35.056889+0000","scrub_duration":0.013778076,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":789,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":2684,"num_read_kb":2204,"num_write":65,"num_write_kb":65,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":903,"num_omap_keys":30,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"55'2257","reported_seq":4814,"reported_epoch":55,"state":"active+clean","last_fresh":"2023-10-03T17:45:00.067330+0000","last_change":"2023-10-03T17:40:13.376519+0000","last_active":"2023-10-03T17:45:00.067330+0000","last_peered":"2023-10-03T17:45:00.067330+0000","last_clean":"2023-10-03T17:45:00.067330+0000","last_became_active":"2023-10-03T17:34:42.271201+0000","last_became_peered":"2023-10-03T17:34:42.271201+0000","last_unstale":"2023-10-03T17:45:00.067330+0000","last_undegraded":"2023-10-03T17:45:00.067330+0000","last_fullsized":"2023-10-03T17:45:00.067330+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'1129","last_scrub_stamp":"2023-10-03T17:40:13.376332+0000","last_deep_scrub":"46'1129","last_deep_scrub_stamp":"2023-10-03T17:40:13.376332+0000","last_clean_scrub_stamp":"2023-10-03T17:40:13.376332+0000","objects_scrubbed":11,"log_size":2257,"log_dups_size":0,"ondisk_log_size":2257,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T18:59:17.088585+0000","scrub_duration":0.015998675,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":724,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":5521,"num_read_kb":4742,"num_write":138,"num_write_kb":138,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2912,"num_omap_keys":93,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"55'28","reported_seq":113,"reported_epoch":55,"state":"active+clean","last_fresh":"2023-10-03T17:44:01.292188+0000","last_change":"2023-10-03T17:40:25.496010+0000","last_active":"2023-10-03T17:44:01.292188+0000","last_peered":"2023-10-03T17:44:01.292188+0000","last_clean":"2023-10-03T17:44:01.292188+0000","last_became_active":"2023-10-03T17:34:42.276941+0000","last_became_peered":"2023-10-03T17:34:42.276941+0000","last_unstale":"2023-10-03T17:44:01.292188+0000","last_undegraded":"2023-10-03T17:44:01.292188+0000","last_fullsized":"2023-10-03T17:44:01.292188+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'14","last_scrub_stamp":"2023-10-03T17:40:25.495827+0000","last_deep_scrub":"46'14","last_deep_scrub_stamp":"2023-10-03T17:40:25.495827+0000","last_clean_scrub_stamp":"2023-10-03T17:40:25.495827+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:05:44.939663+0000","scrub_duration":0.0133078,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":957,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":21,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"55'1808","reported_seq":3686,"reported_epoch":55,"state":"active+clean","last_fresh":"2023-10-03T17:45:00.063054+0000","last_change":"2023-10-03T17:40:26.513902+0000","last_active":"2023-10-03T17:45:00.063054+0000","last_peered":"2023-10-03T17:45:00.063054+0000","last_clean":"2023-10-03T17:45:00.063054+0000","last_became_active":"2023-10-03T17:34:42.276459+0000","last_became_peered":"2023-10-03T17:34:42.276459+0000","last_unstale":"2023-10-03T17:45:00.063054+0000","last_undegraded":"2023-10-03T17:45:00.063054+0000","last_fullsized":"2023-10-03T17:45:00.063054+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'964","last_scrub_stamp":"2023-10-03T17:40:26.513729+0000","last_deep_scrub":"46'964","last_deep_scrub_stamp":"2023-10-03T17:40:26.513729+0000","last_clean_scrub_stamp":"2023-10-03T17:40:26.513729+0000","objects_scrubbed":10,"log_size":1808,"log_dups_size":0,"ondisk_log_size":1808,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:41:33.182751+0000","scrub_duration":0.018190280999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":334,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":4056,"num_read_kb":3379,"num_write":94,"num_write_kb":94,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1731,"num_omap_keys":80,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"55'71874","reported_seq":131001,"reported_epoch":55,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T17:44:59.693551+0000","last_change":"2023-10-03T17:40:11.716583+0000","last_active":"2023-10-03T17:44:59.693551+0000","last_peered":"2023-10-03T17:44:59.693551+0000","last_clean":"2023-10-03T17:44:59.693551+0000","last_became_active":"2023-10-03T17:34:48.310701+0000","last_became_peered":"2023-10-03T17:34:48.310701+0000","last_unstale":"2023-10-03T17:44:59.693551+0000","last_undegraded":"2023-10-03T17:44:59.693551+0000","last_fullsized":"2023-10-03T17:44:59.693551+0000","mapping_epoch":16,"log_start":"53'61866","ondisk_log_start":"53'61866","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_clean_scrub_stamp":"2023-10-03T17:34:47.281095+0000","objects_scrubbed":2650,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 287s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040768800000000002,"stat_sum":{"num_bytes":5523116032,"num_objects":6549,"num_object_clones":5229,"num_object_copies":13098,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6549,"num_whiteouts":0,"num_read":64015,"num_read_kb":237796,"num_write":65242,"num_write_kb":260968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"55'1768","reported_seq":3242,"reported_epoch":55,"state":"active+clean","last_fresh":"2023-10-03T17:44:57.973710+0000","last_change":"2023-10-03T17:40:08.675119+0000","last_active":"2023-10-03T17:44:57.973710+0000","last_peered":"2023-10-03T17:44:57.973710+0000","last_clean":"2023-10-03T17:44:57.973710+0000","last_became_active":"2023-10-03T17:34:42.270625+0000","last_became_peered":"2023-10-03T17:34:42.270625+0000","last_unstale":"2023-10-03T17:44:57.973710+0000","last_undegraded":"2023-10-03T17:44:57.973710+0000","last_fullsized":"2023-10-03T17:44:57.973710+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'841","last_scrub_stamp":"2023-10-03T17:40:08.674910+0000","last_deep_scrub":"46'841","last_deep_scrub_stamp":"2023-10-03T17:40:08.674910+0000","last_clean_scrub_stamp":"2023-10-03T17:40:08.674910+0000","objects_scrubbed":11,"log_size":1768,"log_dups_size":0,"ondisk_log_size":1768,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T21:05:42.790614+0000","scrub_duration":0.039189555000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":857,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":3520,"num_read_kb":2901,"num_write":103,"num_write_kb":103,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2564,"num_omap_keys":60,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"55'67131","reported_seq":122963,"reported_epoch":55,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T17:45:00.029647+0000","last_change":"2023-10-03T17:43:28.494717+0000","last_active":"2023-10-03T17:45:00.029647+0000","last_peered":"2023-10-03T17:45:00.029647+0000","last_clean":"2023-10-03T17:45:00.029647+0000","last_became_active":"2023-10-03T17:34:48.310555+0000","last_became_peered":"2023-10-03T17:34:48.310555+0000","last_unstale":"2023-10-03T17:45:00.029647+0000","last_undegraded":"2023-10-03T17:45:00.029647+0000","last_fullsized":"2023-10-03T17:45:00.029647+0000","mapping_epoch":16,"log_start":"53'57049","ondisk_log_start":"53'57049","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_clean_scrub_stamp":"2023-10-03T17:34:47.281095+0000","objects_scrubbed":1381,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 91s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086289699999999995,"stat_sum":{"num_bytes":5199409152,"num_objects":6171,"num_object_clones":4929,"num_object_copies":12342,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6171,"num_whiteouts":0,"num_read":60532,"num_read_kb":224316,"num_write":61113,"num_write_kb":244452,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"55'92","reported_seq":1190,"reported_epoch":55,"state":"active+clean","last_fresh":"2023-10-03T17:44:55.630537+0000","last_change":"2023-10-03T17:40:17.641493+0000","last_active":"2023-10-03T17:44:55.630537+0000","last_peered":"2023-10-03T17:44:55.630537+0000","last_clean":"2023-10-03T17:44:55.630537+0000","last_became_active":"2023-10-03T17:34:42.278196+0000","last_became_peered":"2023-10-03T17:34:42.278196+0000","last_unstale":"2023-10-03T17:44:55.630537+0000","last_undegraded":"2023-10-03T17:44:55.630537+0000","last_fullsized":"2023-10-03T17:44:55.630537+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'30","last_scrub_stamp":"2023-10-03T17:40:17.640969+0000","last_deep_scrub":"46'30","last_deep_scrub_stamp":"2023-10-03T17:40:17.640969+0000","last_clean_scrub_stamp":"2023-10-03T17:40:17.640969+0000","objects_scrubbed":6,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:04:02.954896+0000","scrub_duration":0.071054565,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":420,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":2611,"num_read_kb":2152,"num_write":57,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":903,"num_omap_keys":30,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"55'68691","reported_seq":125821,"reported_epoch":55,"state":"active+clean","last_fresh":"2023-10-03T17:44:59.691945+0000","last_change":"2023-10-03T17:43:26.193513+0000","last_active":"2023-10-03T17:44:59.691945+0000","last_peered":"2023-10-03T17:44:59.691945+0000","last_clean":"2023-10-03T17:44:59.691945+0000","last_became_active":"2023-10-03T17:34:48.310912+0000","last_became_peered":"2023-10-03T17:34:48.310912+0000","last_unstale":"2023-10-03T17:44:59.691945+0000","last_undegraded":"2023-10-03T17:44:59.691945+0000","last_fullsized":"2023-10-03T17:44:59.691945+0000","mapping_epoch":16,"log_start":"53'58650","ondisk_log_start":"53'58650","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'58196","last_scrub_stamp":"2023-10-03T17:43:26.193075+0000","last_deep_scrub":"53'58196","last_deep_scrub_stamp":"2023-10-03T17:43:26.193075+0000","last_clean_scrub_stamp":"2023-10-03T17:43:26.193075+0000","objects_scrubbed":2326,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":196,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:41:46.955185+0000","scrub_duration":195.44739730800001,"objects_trimmed":0,"snaptrim_duration":0.00045661900000000001,"stat_sum":{"num_bytes":5318873088,"num_objects":6301,"num_object_clones":5030,"num_object_copies":12602,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6301,"num_whiteouts":0,"num_read":61862,"num_read_kb":230620,"num_write":62639,"num_write_kb":250556,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"55'49","reported_seq":477,"reported_epoch":55,"state":"active+clean","last_fresh":"2023-10-03T17:44:50.528465+0000","last_change":"2023-10-03T17:40:10.708344+0000","last_active":"2023-10-03T17:44:50.528465+0000","last_peered":"2023-10-03T17:44:50.528465+0000","last_clean":"2023-10-03T17:44:50.528465+0000","last_became_active":"2023-10-03T17:34:42.272306+0000","last_became_peered":"2023-10-03T17:34:42.272306+0000","last_unstale":"2023-10-03T17:44:50.528465+0000","last_undegraded":"2023-10-03T17:44:50.528465+0000","last_fullsized":"2023-10-03T17:44:50.528465+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'20","last_scrub_stamp":"2023-10-03T17:40:10.708060+0000","last_deep_scrub":"46'20","last_deep_scrub_stamp":"2023-10-03T17:40:10.708060+0000","last_clean_scrub_stamp":"2023-10-03T17:40:10.708060+0000","objects_scrubbed":7,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:03:09.097252+0000","scrub_duration":0.042279491000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":754,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":905,"num_read_kb":728,"num_write":33,"num_write_kb":33,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":301,"num_omap_keys":10,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"55'70647","reported_seq":128118,"reported_epoch":55,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T17:45:00.038396+0000","last_change":"2023-10-03T17:40:29.562114+0000","last_active":"2023-10-03T17:45:00.038396+0000","last_peered":"2023-10-03T17:45:00.038396+0000","last_clean":"2023-10-03T17:45:00.038396+0000","last_became_active":"2023-10-03T17:34:48.310612+0000","last_became_peered":"2023-10-03T17:34:48.310612+0000","last_unstale":"2023-10-03T17:45:00.038396+0000","last_undegraded":"2023-10-03T17:45:00.038396+0000","last_fullsized":"2023-10-03T17:45:00.038396+0000","mapping_epoch":16,"log_start":"53'60569","ondisk_log_start":"53'60569","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_clean_scrub_stamp":"2023-10-03T17:34:47.281095+0000","objects_scrubbed":3056,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 270s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022375120000000001,"stat_sum":{"num_bytes":5386248211,"num_objects":6400,"num_object_clones":5112,"num_object_copies":12800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6400,"num_whiteouts":0,"num_read":62249,"num_read_kb":231496,"num_write":63727,"num_write_kb":254902,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"55'16","reported_seq":94,"reported_epoch":55,"state":"active+clean","last_fresh":"2023-10-03T17:44:01.337431+0000","last_change":"2023-10-03T17:40:24.472734+0000","last_active":"2023-10-03T17:44:01.337431+0000","last_peered":"2023-10-03T17:44:01.337431+0000","last_clean":"2023-10-03T17:44:01.337431+0000","last_became_active":"2023-10-03T17:34:42.270194+0000","last_became_peered":"2023-10-03T17:34:42.270194+0000","last_unstale":"2023-10-03T17:44:01.337431+0000","last_undegraded":"2023-10-03T17:44:01.337431+0000","last_fullsized":"2023-10-03T17:44:01.337431+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'8","last_scrub_stamp":"2023-10-03T17:40:24.472573+0000","last_deep_scrub":"46'8","last_deep_scrub_stamp":"2023-10-03T17:40:24.472573+0000","last_clean_scrub_stamp":"2023-10-03T17:40:24.472573+0000","objects_scrubbed":4,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T18:28:04.735033+0000","scrub_duration":0.024990637999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":537,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":26,"num_read_kb":8,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"46'99","reported_seq":197,"reported_epoch":55,"state":"active+clean","last_fresh":"2023-10-03T17:44:01.316998+0000","last_change":"2023-10-03T17:40:10.452341+0000","last_active":"2023-10-03T17:44:01.316998+0000","last_peered":"2023-10-03T17:44:01.316998+0000","last_clean":"2023-10-03T17:44:01.316998+0000","last_became_active":"2023-10-03T17:34:39.502393+0000","last_became_peered":"2023-10-03T17:34:39.502393+0000","last_unstale":"2023-10-03T17:44:01.316998+0000","last_undegraded":"2023-10-03T17:44:01.316998+0000","last_fullsized":"2023-10-03T17:44:01.316998+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'99","last_scrub_stamp":"2023-10-03T17:40:10.449099+0000","last_deep_scrub":"46'99","last_deep_scrub_stamp":"2023-10-03T17:40:10.449099+0000","last_clean_scrub_stamp":"2023-10-03T17:40:10.449099+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T20:18:04.958666+0000","scrub_duration":0.043225012,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":21427646483,"num_objects":25421,"num_object_clones":20300,"num_object_copies":50842,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25421,"num_whiteouts":0,"num_read":248658,"num_read_kb":924228,"num_write":252721,"num_write_kb":1010878,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2069766144,"data_stored":8386240550,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40209,"ondisk_log_size":40209,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":5372,"num_objects":143,"num_object_clones":0,"num_object_copies":286,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":143,"num_whiteouts":0,"num_read":19362,"num_read_kb":16126,"num_write":523,"num_write_kb":523,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":983040,"data_stored":10744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7366,"ondisk_log_size":7366,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":7460,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738499,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2162672,"kb_used_data":1010792,"kb_used_omap":1,"kb_used_meta":1151870,"kb_avail":92209168,"statfs":{"total":96636764160,"available":94422188032,"internally_reserved":0,"allocated":1035051008,"data_stored":4192865903,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1179515334},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,4,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 17:44:55 2023","interfaces":[{"interface":"back","average":{"1min":0.25800000000000001,"5min":0.40500000000000003,"15min":0.45700000000000002},"min":{"1min":0.16500000000000001,"5min":0.158,"15min":0.14299999999999999},"max":{"1min":0.45600000000000002,"5min":4.1539999999999999,"15min":4.4640000000000004},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.30299999999999999,"5min":0.44,"15min":0.47099999999999997},"min":{"1min":0.20999999999999999,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":0.54800000000000004,"5min":4.4180000000000001,"15min":4.4180000000000001},"last":0.40500000000000003}]},{"osd":1,"last update":"Tue Oct 3 17:44:55 2023","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.55100000000000005,"15min":0.52500000000000002},"min":{"1min":0.19800000000000001,"5min":0.17599999999999999,"15min":0.16600000000000001},"max":{"1min":8.1059999999999999,"5min":8.1059999999999999,"15min":8.1059999999999999},"last":0.33100000000000002},{"interface":"front","average":{"1min":0.34599999999999997,"5min":0.42999999999999999,"15min":0.48299999999999998},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.64400000000000002,"5min":3.9740000000000002,"15min":3.9740000000000002},"last":0.439}]}]},{"osd":1,"up_from":8,"seq":34359738497,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1104068,"kb_used_data":499772,"kb_used_omap":1,"kb_used_meta":604286,"kb_avail":93267772,"statfs":{"total":96636764160,"available":95506198528,"internally_reserved":0,"allocated":511766528,"data_stored":2075549404,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":618789318},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,2,0,2,1,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 17:44:07 2023","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.33200000000000002,"15min":0.42599999999999999},"min":{"1min":0.20899999999999999,"5min":0.17100000000000001,"15min":0.161},"max":{"1min":1.341,"5min":1.8120000000000001,"15min":4.1879999999999997},"last":0.186},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.433,"15min":0.46400000000000002},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":4.5759999999999996,"5min":4.5759999999999996,"15min":4.5759999999999996},"last":0.19700000000000001}]},{"osd":2,"last update":"Tue Oct 3 17:44:07 2023","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.39200000000000002,"15min":0.45200000000000001},"min":{"1min":0.20999999999999999,"5min":0.17799999999999999,"15min":0.159},"max":{"1min":2.6110000000000002,"5min":3.0390000000000001,"15min":3.0390000000000001},"last":0.221},{"interface":"front","average":{"1min":0.34499999999999997,"5min":0.35799999999999998,"15min":0.45800000000000002},"min":{"1min":0.251,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":0.60099999999999998,"5min":3.0699999999999998,"15min":3.0699999999999998},"last":0.16400000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738497,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1124264,"kb_used_data":511648,"kb_used_omap":1,"kb_used_meta":612606,"kb_avail":93247576,"statfs":{"total":96636764160,"available":95485517824,"internally_reserved":0,"allocated":523927552,"data_stored":2117646362,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":627308998},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,8,0,3,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 17:44:52 2023","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.45100000000000001,"15min":0.42699999999999999},"min":{"1min":0.185,"5min":0.154,"15min":0.154},"max":{"1min":1.337,"5min":5.7649999999999997,"15min":5.7649999999999997},"last":0.27900000000000003},{"interface":"front","average":{"1min":0.311,"5min":0.51700000000000002,"15min":0.46999999999999997},"min":{"1min":0.191,"5min":0.19,"15min":0.17699999999999999},"max":{"1min":0.59099999999999997,"5min":9.4770000000000003,"15min":9.4770000000000003},"last":0.35899999999999999}]},{"osd":2,"last update":"Tue Oct 3 17:44:52 2023","interfaces":[{"interface":"back","average":{"1min":0.32400000000000001,"5min":0.53200000000000003,"15min":0.52600000000000002},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.44900000000000001,"5min":9.4149999999999991,"15min":9.4149999999999991},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.434,"5min":0.42299999999999999,"15min":0.48399999999999999},"min":{"1min":0.19700000000000001,"5min":0.189,"15min":0.189},"max":{"1min":1.9359999999999999,"5min":3.8100000000000001,"15min":3.8100000000000001},"last":5.218}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":335872,"data_stored":3661,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":5372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":155648,"data_stored":1711,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":523804672,"data_stored":2117906432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":511340544,"data_stored":2075475987,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1034620928,"data_stored":4192858131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-03T17:45:00.958 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-03T17:34:47.281095+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=34, tm_sec=47, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=40, tm_sec=6, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T17:45:00.958 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-03T17:34:47.281095+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=34, tm_sec=47, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=40, tm_sec=6, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T17:45:00.958 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-03T17:34:47.281095+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=34, tm_sec=47, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=40, tm_sec=6, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T17:45:00.958 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T17:45:20.048 INFO:tasks.ceph.osd.0.smithi111.stderr:2023-10-03T17:45:20.048+0000 7ffbac289640 -1 reset not still connected to 0x5563a838e1a0 2023-10-03T17:45:20.960 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T17:45:21.545 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-03T17:45:21.546 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-10-03T17:45:21.573 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":381,"stamp":"2023-10-03T17:45:20.095872+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":21461853999,"num_objects":28350,"num_object_clones":23084,"num_object_copies":56700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28350,"num_whiteouts":0,"num_read":277610,"num_read_kb":964765,"num_write":259204,"num_write_kb":1035867,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":47925,"ondisk_log_size":47925,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":4658848,"kb_used_data":2070728,"kb_used_omap":4,"kb_used_meta":2588091,"kb_avail":278456672,"statfs":{"total":289910292480,"available":285139632128,"internally_reserved":0,"allocated":2120425472,"data_stored":9611334083,"data_compressed":7460,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4782,"internal_metadata":2650205522},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,2,11,9,9,19,13],"upper_bound":256},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":23994266,"num_objects":1993,"num_object_clones":1995,"num_object_copies":3986,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1993,"num_whiteouts":0,"num_read":5396,"num_read_kb":16236,"num_write":4042,"num_write_kb":16106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001862"},"pg_stats":[{"pgid":"2.7","version":"61'1387","reported_seq":2648,"reported_epoch":61,"state":"active+clean","last_fresh":"2023-10-03T17:45:11.713279+0000","last_change":"2023-10-03T17:40:17.407011+0000","last_active":"2023-10-03T17:45:11.713279+0000","last_peered":"2023-10-03T17:45:11.713279+0000","last_clean":"2023-10-03T17:45:11.713279+0000","last_became_active":"2023-10-03T17:34:42.276624+0000","last_became_peered":"2023-10-03T17:34:42.276624+0000","last_unstale":"2023-10-03T17:45:11.713279+0000","last_undegraded":"2023-10-03T17:45:11.713279+0000","last_fullsized":"2023-10-03T17:45:11.713279+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'673","last_scrub_stamp":"2023-10-03T17:40:17.406813+0000","last_deep_scrub":"46'673","last_deep_scrub_stamp":"2023-10-03T17:40:17.406813+0000","last_clean_scrub_stamp":"2023-10-03T17:40:17.406813+0000","objects_scrubbed":7,"log_size":1387,"log_dups_size":0,"ondisk_log_size":1387,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T21:38:35.056889+0000","scrub_duration":0.013778076,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":638,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":3257,"num_read_kb":2681,"num_write":99,"num_write_kb":88,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":903,"num_omap_keys":30,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"61'2321","reported_seq":5219,"reported_epoch":61,"state":"active+clean","last_fresh":"2023-10-03T17:45:11.712726+0000","last_change":"2023-10-03T17:40:13.376519+0000","last_active":"2023-10-03T17:45:11.712726+0000","last_peered":"2023-10-03T17:45:11.712726+0000","last_clean":"2023-10-03T17:45:11.712726+0000","last_became_active":"2023-10-03T17:34:42.271201+0000","last_became_peered":"2023-10-03T17:34:42.271201+0000","last_unstale":"2023-10-03T17:45:11.712726+0000","last_undegraded":"2023-10-03T17:45:11.712726+0000","last_fullsized":"2023-10-03T17:45:11.712726+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'1129","last_scrub_stamp":"2023-10-03T17:40:13.376332+0000","last_deep_scrub":"46'1129","last_deep_scrub_stamp":"2023-10-03T17:40:13.376332+0000","last_clean_scrub_stamp":"2023-10-03T17:40:13.376332+0000","objects_scrubbed":11,"log_size":2321,"log_dups_size":0,"ondisk_log_size":2321,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T18:59:17.088585+0000","scrub_duration":0.015998675,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":927,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":6655,"num_read_kb":5707,"num_write":193,"num_write_kb":180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2912,"num_omap_keys":93,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"60'30","reported_seq":123,"reported_epoch":61,"state":"active+clean","last_fresh":"2023-10-03T17:45:06.351382+0000","last_change":"2023-10-03T17:40:25.496010+0000","last_active":"2023-10-03T17:45:06.351382+0000","last_peered":"2023-10-03T17:45:06.351382+0000","last_clean":"2023-10-03T17:45:06.351382+0000","last_became_active":"2023-10-03T17:34:42.276941+0000","last_became_peered":"2023-10-03T17:34:42.276941+0000","last_unstale":"2023-10-03T17:45:06.351382+0000","last_undegraded":"2023-10-03T17:45:06.351382+0000","last_fullsized":"2023-10-03T17:45:06.351382+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'14","last_scrub_stamp":"2023-10-03T17:40:25.495827+0000","last_deep_scrub":"46'14","last_deep_scrub_stamp":"2023-10-03T17:40:25.495827+0000","last_clean_scrub_stamp":"2023-10-03T17:40:25.495827+0000","objects_scrubbed":7,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:05:44.939663+0000","scrub_duration":0.0133078,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":956,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":40,"num_read_kb":13,"num_write":23,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"61'1855","reported_seq":3965,"reported_epoch":61,"state":"active+clean","last_fresh":"2023-10-03T17:45:11.714973+0000","last_change":"2023-10-03T17:40:26.513902+0000","last_active":"2023-10-03T17:45:11.714973+0000","last_peered":"2023-10-03T17:45:11.714973+0000","last_clean":"2023-10-03T17:45:11.714973+0000","last_became_active":"2023-10-03T17:34:42.276459+0000","last_became_peered":"2023-10-03T17:34:42.276459+0000","last_unstale":"2023-10-03T17:45:11.714973+0000","last_undegraded":"2023-10-03T17:45:11.714973+0000","last_fullsized":"2023-10-03T17:45:11.714973+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'964","last_scrub_stamp":"2023-10-03T17:40:26.513729+0000","last_deep_scrub":"46'964","last_deep_scrub_stamp":"2023-10-03T17:40:26.513729+0000","last_clean_scrub_stamp":"2023-10-03T17:40:26.513729+0000","objects_scrubbed":10,"log_size":1855,"log_dups_size":0,"ondisk_log_size":1855,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:41:33.182751+0000","scrub_duration":0.018190280999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":283,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":4830,"num_read_kb":4032,"num_write":133,"num_write_kb":123,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1731,"num_omap_keys":80,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"61'74475","reported_seq":134311,"reported_epoch":61,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T17:45:19.888184+0000","last_change":"2023-10-03T17:40:11.716583+0000","last_active":"2023-10-03T17:45:19.888184+0000","last_peered":"2023-10-03T17:45:19.888184+0000","last_clean":"2023-10-03T17:45:19.888184+0000","last_became_active":"2023-10-03T17:34:48.310701+0000","last_became_peered":"2023-10-03T17:34:48.310701+0000","last_unstale":"2023-10-03T17:45:19.888184+0000","last_undegraded":"2023-10-03T17:45:19.888184+0000","last_fullsized":"2023-10-03T17:45:19.888184+0000","mapping_epoch":16,"log_start":"55'64472","ondisk_log_start":"55'64472","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_clean_scrub_stamp":"2023-10-03T17:34:47.281095+0000","objects_scrubbed":2830,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":20,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 308s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040768800000000002,"stat_sum":{"num_bytes":5532647424,"num_objects":7414,"num_object_clones":6094,"num_object_copies":14828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7414,"num_whiteouts":0,"num_read":65562,"num_read_kb":243856,"num_write":66838,"num_write_kb":267352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"61'1837","reported_seq":3517,"reported_epoch":61,"state":"active+clean","last_fresh":"2023-10-03T17:45:19.702973+0000","last_change":"2023-10-03T17:40:08.675119+0000","last_active":"2023-10-03T17:45:19.702973+0000","last_peered":"2023-10-03T17:45:19.702973+0000","last_clean":"2023-10-03T17:45:19.702973+0000","last_became_active":"2023-10-03T17:34:42.270625+0000","last_became_peered":"2023-10-03T17:34:42.270625+0000","last_unstale":"2023-10-03T17:45:19.702973+0000","last_undegraded":"2023-10-03T17:45:19.702973+0000","last_fullsized":"2023-10-03T17:45:19.702973+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'841","last_scrub_stamp":"2023-10-03T17:40:08.674910+0000","last_deep_scrub":"46'841","last_deep_scrub_stamp":"2023-10-03T17:40:08.674910+0000","last_clean_scrub_stamp":"2023-10-03T17:40:08.674910+0000","objects_scrubbed":11,"log_size":1837,"log_dups_size":0,"ondisk_log_size":1837,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T21:05:42.790614+0000","scrub_duration":0.039189555000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":808,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":4361,"num_read_kb":3599,"num_write":144,"num_write_kb":133,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2564,"num_omap_keys":60,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"61'69130","reported_seq":125383,"reported_epoch":61,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T17:45:15.212890+0000","last_change":"2023-10-03T17:43:28.494717+0000","last_active":"2023-10-03T17:45:15.212890+0000","last_peered":"2023-10-03T17:45:15.212890+0000","last_clean":"2023-10-03T17:45:15.212890+0000","last_became_active":"2023-10-03T17:34:48.310555+0000","last_became_peered":"2023-10-03T17:34:48.310555+0000","last_unstale":"2023-10-03T17:45:15.212890+0000","last_undegraded":"2023-10-03T17:45:15.212890+0000","last_fullsized":"2023-10-03T17:45:15.212890+0000","mapping_epoch":16,"log_start":"53'59056","ondisk_log_start":"53'59056","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_clean_scrub_stamp":"2023-10-03T17:34:47.281095+0000","objects_scrubbed":1540,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":20,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 106s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086289699999999995,"stat_sum":{"num_bytes":5206904832,"num_objects":6876,"num_object_clones":5634,"num_object_copies":13752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6876,"num_whiteouts":0,"num_read":61635,"num_read_kb":228636,"num_write":62283,"num_write_kb":249132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"61'112","reported_seq":1363,"reported_epoch":61,"state":"active+clean","last_fresh":"2023-10-03T17:45:16.100354+0000","last_change":"2023-10-03T17:40:17.641493+0000","last_active":"2023-10-03T17:45:16.100354+0000","last_peered":"2023-10-03T17:45:16.100354+0000","last_clean":"2023-10-03T17:45:16.100354+0000","last_became_active":"2023-10-03T17:34:42.278196+0000","last_became_peered":"2023-10-03T17:34:42.278196+0000","last_unstale":"2023-10-03T17:45:16.100354+0000","last_undegraded":"2023-10-03T17:45:16.100354+0000","last_fullsized":"2023-10-03T17:45:16.100354+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'30","last_scrub_stamp":"2023-10-03T17:40:17.640969+0000","last_deep_scrub":"46'30","last_deep_scrub_stamp":"2023-10-03T17:40:17.640969+0000","last_clean_scrub_stamp":"2023-10-03T17:40:17.640969+0000","objects_scrubbed":6,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:04:02.954896+0000","scrub_duration":0.071054565,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":420,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":3239,"num_read_kb":2672,"num_write":86,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":903,"num_omap_keys":30,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"61'71799","reported_seq":129485,"reported_epoch":61,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T17:45:19.881316+0000","last_change":"2023-10-03T17:45:16.313353+0000","last_active":"2023-10-03T17:45:19.881316+0000","last_peered":"2023-10-03T17:45:19.881316+0000","last_clean":"2023-10-03T17:45:19.881316+0000","last_became_active":"2023-10-03T17:34:48.310912+0000","last_became_peered":"2023-10-03T17:34:48.310912+0000","last_unstale":"2023-10-03T17:45:19.881316+0000","last_undegraded":"2023-10-03T17:45:19.881316+0000","last_fullsized":"2023-10-03T17:45:19.881316+0000","mapping_epoch":16,"log_start":"53'61770","ondisk_log_start":"53'61770","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'58196","last_scrub_stamp":"2023-10-03T17:43:26.193075+0000","last_deep_scrub":"53'58196","last_deep_scrub_stamp":"2023-10-03T17:43:26.193075+0000","last_clean_scrub_stamp":"2023-10-03T17:43:26.193075+0000","objects_scrubbed":2326,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":16,"last_scrub_duration":196,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T23:17:09.917848+0000","scrub_duration":195.44739730800001,"objects_trimmed":249,"snaptrim_duration":9.9512050629999997,"stat_sum":{"num_bytes":5329186816,"num_objects":6797,"num_object_clones":5526,"num_object_copies":13594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6797,"num_whiteouts":0,"num_read":63385,"num_read_kb":236608,"num_write":64257,"num_write_kb":257028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"54","length":"4"}]},{"pgid":"2.0","version":"59'58","reported_seq":542,"reported_epoch":61,"state":"active+clean","last_fresh":"2023-10-03T17:45:11.535569+0000","last_change":"2023-10-03T17:40:10.708344+0000","last_active":"2023-10-03T17:45:11.535569+0000","last_peered":"2023-10-03T17:45:11.535569+0000","last_clean":"2023-10-03T17:45:11.535569+0000","last_became_active":"2023-10-03T17:34:42.272306+0000","last_became_peered":"2023-10-03T17:34:42.272306+0000","last_unstale":"2023-10-03T17:45:11.535569+0000","last_undegraded":"2023-10-03T17:45:11.535569+0000","last_fullsized":"2023-10-03T17:45:11.535569+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'20","last_scrub_stamp":"2023-10-03T17:40:10.708060+0000","last_deep_scrub":"46'20","last_deep_scrub_stamp":"2023-10-03T17:40:10.708060+0000","last_clean_scrub_stamp":"2023-10-03T17:40:10.708060+0000","objects_scrubbed":7,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:03:09.097252+0000","scrub_duration":0.042279491000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":704,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":1116,"num_read_kb":903,"num_write":45,"num_write_kb":41,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":301,"num_omap_keys":10,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"61'72673","reported_seq":130589,"reported_epoch":61,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T17:45:15.232013+0000","last_change":"2023-10-03T17:40:29.562114+0000","last_active":"2023-10-03T17:45:15.232013+0000","last_peered":"2023-10-03T17:45:15.232013+0000","last_clean":"2023-10-03T17:45:15.232013+0000","last_became_active":"2023-10-03T17:34:48.310612+0000","last_became_peered":"2023-10-03T17:34:48.310612+0000","last_unstale":"2023-10-03T17:45:15.232013+0000","last_undegraded":"2023-10-03T17:45:15.232013+0000","last_fullsized":"2023-10-03T17:45:15.232013+0000","mapping_epoch":16,"log_start":"53'62577","ondisk_log_start":"53'62577","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_clean_scrub_stamp":"2023-10-03T17:34:47.281095+0000","objects_scrubbed":3176,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":20,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 285s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022375120000000001,"stat_sum":{"num_bytes":5392519187,"num_objects":7118,"num_object_clones":5830,"num_object_copies":14236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7118,"num_whiteouts":0,"num_read":63395,"num_read_kb":235956,"num_write":64946,"num_write_kb":259778,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"59'24","reported_seq":112,"reported_epoch":61,"state":"active+clean","last_fresh":"2023-10-03T17:45:06.351443+0000","last_change":"2023-10-03T17:40:24.472734+0000","last_active":"2023-10-03T17:45:06.351443+0000","last_peered":"2023-10-03T17:45:06.351443+0000","last_clean":"2023-10-03T17:45:06.351443+0000","last_became_active":"2023-10-03T17:34:42.270194+0000","last_became_peered":"2023-10-03T17:34:42.270194+0000","last_unstale":"2023-10-03T17:45:06.351443+0000","last_undegraded":"2023-10-03T17:45:06.351443+0000","last_fullsized":"2023-10-03T17:45:06.351443+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'8","last_scrub_stamp":"2023-10-03T17:40:24.472573+0000","last_deep_scrub":"46'8","last_deep_scrub_stamp":"2023-10-03T17:40:24.472573+0000","last_clean_scrub_stamp":"2023-10-03T17:40:24.472573+0000","objects_scrubbed":4,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T18:28:04.735033+0000","scrub_duration":0.024990637999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":636,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":29,"num_read_kb":11,"num_write":20,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"46'99","reported_seq":203,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-03T17:45:05.340127+0000","last_change":"2023-10-03T17:40:10.452341+0000","last_active":"2023-10-03T17:45:05.340127+0000","last_peered":"2023-10-03T17:45:05.340127+0000","last_clean":"2023-10-03T17:45:05.340127+0000","last_became_active":"2023-10-03T17:34:39.502393+0000","last_became_peered":"2023-10-03T17:34:39.502393+0000","last_unstale":"2023-10-03T17:45:05.340127+0000","last_undegraded":"2023-10-03T17:45:05.340127+0000","last_fullsized":"2023-10-03T17:45:05.340127+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'99","last_scrub_stamp":"2023-10-03T17:40:10.449099+0000","last_deep_scrub":"46'99","last_deep_scrub_stamp":"2023-10-03T17:40:10.449099+0000","last_clean_scrub_stamp":"2023-10-03T17:40:10.449099+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T20:18:04.958666+0000","scrub_duration":0.043225012,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":21461258259,"num_objects":28205,"num_object_clones":23084,"num_object_copies":56410,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28205,"num_whiteouts":0,"num_read":253977,"num_read_kb":945056,"num_write":258324,"num_write_kb":1033290,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2118959104,"data_stored":9632849958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40202,"ondisk_log_size":40202,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":5372,"num_objects":143,"num_object_clones":0,"num_object_copies":286,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":143,"num_whiteouts":0,"num_read":23527,"num_read_kb":19618,"num_write":743,"num_write_kb":683,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":983040,"data_stored":10744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7624,"ondisk_log_size":7624,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":7460,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738503,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2286740,"kb_used_data":1036300,"kb_used_omap":1,"kb_used_meta":1250430,"kb_avail":92085100,"statfs":{"total":96636764160,"available":94295142400,"internally_reserved":0,"allocated":1061171200,"data_stored":4836498794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1280440774},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,1,5,9,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 17:44:55 2023","interfaces":[{"interface":"back","average":{"1min":0.25800000000000001,"5min":0.40500000000000003,"15min":0.45700000000000002},"min":{"1min":0.16500000000000001,"5min":0.158,"15min":0.14299999999999999},"max":{"1min":0.45600000000000002,"5min":4.1539999999999999,"15min":4.4640000000000004},"last":0.217},{"interface":"front","average":{"1min":0.30299999999999999,"5min":0.44,"15min":0.47099999999999997},"min":{"1min":0.20999999999999999,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":0.54800000000000004,"5min":4.4180000000000001,"15min":4.4180000000000001},"last":0.19700000000000001}]},{"osd":1,"last update":"Tue Oct 3 17:44:55 2023","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.55100000000000005,"15min":0.52500000000000002},"min":{"1min":0.19800000000000001,"5min":0.17599999999999999,"15min":0.16600000000000001},"max":{"1min":8.1059999999999999,"5min":8.1059999999999999,"15min":8.1059999999999999},"last":0.33100000000000002},{"interface":"front","average":{"1min":0.34599999999999997,"5min":0.42999999999999999,"15min":0.48299999999999998},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.64400000000000002,"5min":3.9740000000000002,"15min":3.9740000000000002},"last":0.307}]}]},{"osd":1,"up_from":8,"seq":34359738500,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1180444,"kb_used_data":509524,"kb_used_omap":1,"kb_used_meta":670910,"kb_avail":93191396,"statfs":{"total":96636764160,"available":95427989504,"internally_reserved":0,"allocated":521752576,"data_stored":2365816230,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":687012294},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,6,4,3,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 17:45:08 2023","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.32900000000000001,"15min":0.41399999999999998},"min":{"1min":0.17199999999999999,"5min":0.17100000000000001,"15min":0.161},"max":{"1min":2.3809999999999998,"5min":2.3809999999999998,"15min":4.1879999999999997},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.27500000000000002,"5min":0.41399999999999998,"15min":0.44600000000000001},"min":{"1min":0.17799999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.46200000000000002,"5min":4.5759999999999996,"15min":4.5759999999999996},"last":0.21199999999999999}]},{"osd":2,"last update":"Tue Oct 3 17:45:08 2023","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.39000000000000001,"15min":0.44},"min":{"1min":0.214,"5min":0.187,"15min":0.159},"max":{"1min":1.3100000000000001,"5min":3.0390000000000001,"15min":3.0390000000000001},"last":0.33900000000000002},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.35299999999999998,"15min":0.443},"min":{"1min":0.16400000000000001,"5min":0.16400000000000001,"15min":0.16400000000000001},"max":{"1min":1.337,"5min":3.0699999999999998,"15min":3.0699999999999998},"last":0.38900000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738501,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1191664,"kb_used_data":524904,"kb_used_omap":1,"kb_used_meta":666750,"kb_avail":93180176,"statfs":{"total":96636764160,"available":95416500224,"internally_reserved":0,"allocated":537501696,"data_stored":2409019059,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":682752454},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,9,2,0,7,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 17:44:52 2023","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.45100000000000001,"15min":0.42699999999999999},"min":{"1min":0.185,"5min":0.154,"15min":0.154},"max":{"1min":1.337,"5min":5.7649999999999997,"15min":5.7649999999999997},"last":0.27200000000000002},{"interface":"front","average":{"1min":0.311,"5min":0.51700000000000002,"15min":0.46999999999999997},"min":{"1min":0.191,"5min":0.19,"15min":0.17699999999999999},"max":{"1min":0.59099999999999997,"5min":9.4770000000000003,"15min":9.4770000000000003},"last":0.34699999999999998}]},{"osd":2,"last update":"Tue Oct 3 17:44:52 2023","interfaces":[{"interface":"back","average":{"1min":0.32400000000000001,"5min":0.53200000000000003,"15min":0.52600000000000002},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.44900000000000001,"5min":9.4149999999999991,"15min":9.4149999999999991},"last":0.311},{"interface":"front","average":{"1min":0.434,"5min":0.42299999999999999,"15min":0.48399999999999999},"min":{"1min":0.19700000000000001,"5min":0.189,"15min":0.189},"max":{"1min":1.9359999999999999,"5min":3.8100000000000001,"15min":3.8100000000000001},"last":0.36599999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":3612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":5372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":159744,"data_stored":1760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":536969216,"data_stored":2414866432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":521125888,"data_stored":2374635539,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1060864000,"data_stored":4843347987,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-03T17:45:21.575 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-03T17:34:47.281095+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=34, tm_sec=47, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=40, tm_sec=6, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T17:45:21.575 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-03T17:34:47.281095+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=34, tm_sec=47, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=40, tm_sec=6, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T17:45:21.576 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-03T17:34:47.281095+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=34, tm_sec=47, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=40, tm_sec=6, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T17:45:21.576 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T17:45:26.843 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:45:26.844+0000 7f09e4851640 -1 reset not still connected to 0x557d76621040 2023-10-03T17:45:26.843 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:45:26.844+0000 7f09e4851640 -1 reset not still connected to 0x557d7664e750 2023-10-03T17:45:29.996 INFO:tasks.ceph.mgr.x.smithi111.stderr:2023-10-03T17:45:29.996+0000 7fe48bb97640 -1 calc_pg_upmaps abort due to max <= 0 2023-10-03T17:45:41.576 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T17:45:42.121 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-03T17:45:42.122 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-10-03T17:45:42.166 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":392,"stamp":"2023-10-03T17:45:42.099749+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":21503657775,"num_objects":29374,"num_object_clones":24108,"num_object_copies":58748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29374,"num_whiteouts":0,"num_read":285032,"num_read_kb":993973,"num_write":266731,"num_write_kb":1065975,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":48126,"ondisk_log_size":48126,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":4802976,"kb_used_data":2127428,"kb_used_omap":4,"kb_used_meta":2675515,"kb_avail":278312544,"statfs":{"total":289910292480,"available":284992045056,"internally_reserved":0,"allocated":2178486272,"data_stored":10072281539,"data_compressed":7460,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4782,"internal_metadata":2739727698},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,11,15,8,13,9,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":23912448,"num_objects":301,"num_object_clones":301,"num_object_copies":602,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":301,"num_whiteouts":0,"num_read":4314,"num_read_kb":16980,"num_write":4287,"num_write_kb":17148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002139"},"pg_stats":[{"pgid":"2.7","version":"61'1433","reported_seq":2710,"reported_epoch":61,"state":"active+clean","last_fresh":"2023-10-03T17:45:37.500117+0000","last_change":"2023-10-03T17:40:17.407011+0000","last_active":"2023-10-03T17:45:37.500117+0000","last_peered":"2023-10-03T17:45:37.500117+0000","last_clean":"2023-10-03T17:45:37.500117+0000","last_became_active":"2023-10-03T17:34:42.276624+0000","last_became_peered":"2023-10-03T17:34:42.276624+0000","last_unstale":"2023-10-03T17:45:37.500117+0000","last_undegraded":"2023-10-03T17:45:37.500117+0000","last_fullsized":"2023-10-03T17:45:37.500117+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'673","last_scrub_stamp":"2023-10-03T17:40:17.406813+0000","last_deep_scrub":"46'673","last_deep_scrub_stamp":"2023-10-03T17:40:17.406813+0000","last_clean_scrub_stamp":"2023-10-03T17:40:17.406813+0000","objects_scrubbed":7,"log_size":1433,"log_dups_size":0,"ondisk_log_size":1433,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T21:38:35.056889+0000","scrub_duration":0.013778076,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":638,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":3257,"num_read_kb":2681,"num_write":99,"num_write_kb":88,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":903,"num_omap_keys":30,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"61'2396","reported_seq":5320,"reported_epoch":61,"state":"active+clean","last_fresh":"2023-10-03T17:45:37.505647+0000","last_change":"2023-10-03T17:40:13.376519+0000","last_active":"2023-10-03T17:45:37.505647+0000","last_peered":"2023-10-03T17:45:37.505647+0000","last_clean":"2023-10-03T17:45:37.505647+0000","last_became_active":"2023-10-03T17:34:42.271201+0000","last_became_peered":"2023-10-03T17:34:42.271201+0000","last_unstale":"2023-10-03T17:45:37.505647+0000","last_undegraded":"2023-10-03T17:45:37.505647+0000","last_fullsized":"2023-10-03T17:45:37.505647+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'1129","last_scrub_stamp":"2023-10-03T17:40:13.376332+0000","last_deep_scrub":"46'1129","last_deep_scrub_stamp":"2023-10-03T17:40:13.376332+0000","last_clean_scrub_stamp":"2023-10-03T17:40:13.376332+0000","objects_scrubbed":11,"log_size":2396,"log_dups_size":0,"ondisk_log_size":2396,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T18:59:17.088585+0000","scrub_duration":0.015998675,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":927,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":6655,"num_read_kb":5707,"num_write":193,"num_write_kb":180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2912,"num_omap_keys":93,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"60'30","reported_seq":123,"reported_epoch":61,"state":"active+clean","last_fresh":"2023-10-03T17:45:06.351382+0000","last_change":"2023-10-03T17:40:25.496010+0000","last_active":"2023-10-03T17:45:06.351382+0000","last_peered":"2023-10-03T17:45:06.351382+0000","last_clean":"2023-10-03T17:45:06.351382+0000","last_became_active":"2023-10-03T17:34:42.276941+0000","last_became_peered":"2023-10-03T17:34:42.276941+0000","last_unstale":"2023-10-03T17:45:06.351382+0000","last_undegraded":"2023-10-03T17:45:06.351382+0000","last_fullsized":"2023-10-03T17:45:06.351382+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'14","last_scrub_stamp":"2023-10-03T17:40:25.495827+0000","last_deep_scrub":"46'14","last_deep_scrub_stamp":"2023-10-03T17:40:25.495827+0000","last_clean_scrub_stamp":"2023-10-03T17:40:25.495827+0000","objects_scrubbed":7,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:05:44.939663+0000","scrub_duration":0.0133078,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":956,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":40,"num_read_kb":13,"num_write":23,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"61'1916","reported_seq":4046,"reported_epoch":61,"state":"active+clean","last_fresh":"2023-10-03T17:45:37.497536+0000","last_change":"2023-10-03T17:40:26.513902+0000","last_active":"2023-10-03T17:45:37.497536+0000","last_peered":"2023-10-03T17:45:37.497536+0000","last_clean":"2023-10-03T17:45:37.497536+0000","last_became_active":"2023-10-03T17:34:42.276459+0000","last_became_peered":"2023-10-03T17:34:42.276459+0000","last_unstale":"2023-10-03T17:45:37.497536+0000","last_undegraded":"2023-10-03T17:45:37.497536+0000","last_fullsized":"2023-10-03T17:45:37.497536+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'964","last_scrub_stamp":"2023-10-03T17:40:26.513729+0000","last_deep_scrub":"46'964","last_deep_scrub_stamp":"2023-10-03T17:40:26.513729+0000","last_clean_scrub_stamp":"2023-10-03T17:40:26.513729+0000","objects_scrubbed":10,"log_size":1916,"log_dups_size":0,"ondisk_log_size":1916,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:41:33.182751+0000","scrub_duration":0.018190280999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":283,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":4830,"num_read_kb":4032,"num_write":133,"num_write_kb":123,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1731,"num_omap_keys":80,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"61'76797","reported_seq":138054,"reported_epoch":61,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T17:45:40.084769+0000","last_change":"2023-10-03T17:40:11.716583+0000","last_active":"2023-10-03T17:45:40.084769+0000","last_peered":"2023-10-03T17:45:40.084769+0000","last_clean":"2023-10-03T17:45:40.084769+0000","last_became_active":"2023-10-03T17:34:48.310701+0000","last_became_peered":"2023-10-03T17:34:48.310701+0000","last_unstale":"2023-10-03T17:45:40.084769+0000","last_undegraded":"2023-10-03T17:45:40.084769+0000","last_fullsized":"2023-10-03T17:45:40.084769+0000","mapping_epoch":16,"log_start":"55'66776","ondisk_log_start":"55'66776","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_clean_scrub_stamp":"2023-10-03T17:34:47.281095+0000","objects_scrubbed":3072,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":20,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 328s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040768800000000002,"stat_sum":{"num_bytes":5541810176,"num_objects":7738,"num_object_clones":6418,"num_object_copies":15476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7738,"num_whiteouts":0,"num_read":67292,"num_read_kb":250672,"num_write":68636,"num_write_kb":274544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"61'1882","reported_seq":3578,"reported_epoch":61,"state":"active+clean","last_fresh":"2023-10-03T17:45:38.266160+0000","last_change":"2023-10-03T17:40:08.675119+0000","last_active":"2023-10-03T17:45:38.266160+0000","last_peered":"2023-10-03T17:45:38.266160+0000","last_clean":"2023-10-03T17:45:38.266160+0000","last_became_active":"2023-10-03T17:34:42.270625+0000","last_became_peered":"2023-10-03T17:34:42.270625+0000","last_unstale":"2023-10-03T17:45:38.266160+0000","last_undegraded":"2023-10-03T17:45:38.266160+0000","last_fullsized":"2023-10-03T17:45:38.266160+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'841","last_scrub_stamp":"2023-10-03T17:40:08.674910+0000","last_deep_scrub":"46'841","last_deep_scrub_stamp":"2023-10-03T17:40:08.674910+0000","last_clean_scrub_stamp":"2023-10-03T17:40:08.674910+0000","objects_scrubbed":11,"log_size":1882,"log_dups_size":0,"ondisk_log_size":1882,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T21:05:42.790614+0000","scrub_duration":0.039189555000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":808,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":4361,"num_read_kb":3599,"num_write":144,"num_write_kb":133,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2564,"num_omap_keys":60,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"61'71889","reported_seq":129675,"reported_epoch":61,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T17:45:40.398843+0000","last_change":"2023-10-03T17:43:28.494717+0000","last_active":"2023-10-03T17:45:40.398843+0000","last_peered":"2023-10-03T17:45:40.398843+0000","last_clean":"2023-10-03T17:45:40.398843+0000","last_became_active":"2023-10-03T17:34:48.310555+0000","last_became_peered":"2023-10-03T17:34:48.310555+0000","last_unstale":"2023-10-03T17:45:40.398843+0000","last_undegraded":"2023-10-03T17:45:40.398843+0000","last_fullsized":"2023-10-03T17:45:40.398843+0000","mapping_epoch":16,"log_start":"55'61859","ondisk_log_start":"55'61859","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_clean_scrub_stamp":"2023-10-03T17:34:47.281095+0000","objects_scrubbed":1990,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":20,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 131s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086289699999999995,"stat_sum":{"num_bytes":5217648640,"num_objects":7285,"num_object_clones":6043,"num_object_copies":14570,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7285,"num_whiteouts":0,"num_read":63555,"num_read_kb":236188,"num_write":64265,"num_write_kb":257060,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"61'112","reported_seq":1375,"reported_epoch":61,"state":"active+clean","last_fresh":"2023-10-03T17:45:36.779086+0000","last_change":"2023-10-03T17:40:17.641493+0000","last_active":"2023-10-03T17:45:36.779086+0000","last_peered":"2023-10-03T17:45:36.779086+0000","last_clean":"2023-10-03T17:45:36.779086+0000","last_became_active":"2023-10-03T17:34:42.278196+0000","last_became_peered":"2023-10-03T17:34:42.278196+0000","last_unstale":"2023-10-03T17:45:36.779086+0000","last_undegraded":"2023-10-03T17:45:36.779086+0000","last_fullsized":"2023-10-03T17:45:36.779086+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'30","last_scrub_stamp":"2023-10-03T17:40:17.640969+0000","last_deep_scrub":"46'30","last_deep_scrub_stamp":"2023-10-03T17:40:17.640969+0000","last_clean_scrub_stamp":"2023-10-03T17:40:17.640969+0000","objects_scrubbed":6,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:04:02.954896+0000","scrub_duration":0.071054565,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":420,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":3239,"num_read_kb":2672,"num_write":86,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":903,"num_omap_keys":30,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"61'74712","reported_seq":133599,"reported_epoch":61,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T17:45:40.032863+0000","last_change":"2023-10-03T17:45:39.332864+0000","last_active":"2023-10-03T17:45:40.032863+0000","last_peered":"2023-10-03T17:45:40.032863+0000","last_clean":"2023-10-03T17:45:40.032863+0000","last_became_active":"2023-10-03T17:34:48.310912+0000","last_became_peered":"2023-10-03T17:34:48.310912+0000","last_unstale":"2023-10-03T17:45:40.032863+0000","last_undegraded":"2023-10-03T17:45:40.032863+0000","last_fullsized":"2023-10-03T17:45:40.032863+0000","mapping_epoch":16,"log_start":"55'64682","ondisk_log_start":"55'64682","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'58196","last_scrub_stamp":"2023-10-03T17:43:26.193075+0000","last_deep_scrub":"53'58196","last_deep_scrub_stamp":"2023-10-03T17:43:26.193075+0000","last_clean_scrub_stamp":"2023-10-03T17:43:26.193075+0000","objects_scrubbed":2326,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":8,"last_scrub_duration":196,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T23:17:09.917848+0000","scrub_duration":195.44739730800001,"objects_trimmed":703,"snaptrim_duration":32.970748725999997,"stat_sum":{"num_bytes":5338468352,"num_objects":6651,"num_object_clones":5380,"num_object_copies":13302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6651,"num_whiteouts":0,"num_read":65078,"num_read_kb":243268,"num_write":65946,"num_write_kb":263784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"54","length":"c"}]},{"pgid":"2.0","version":"59'58","reported_seq":546,"reported_epoch":61,"state":"active+clean","last_fresh":"2023-10-03T17:45:31.456943+0000","last_change":"2023-10-03T17:40:10.708344+0000","last_active":"2023-10-03T17:45:31.456943+0000","last_peered":"2023-10-03T17:45:31.456943+0000","last_clean":"2023-10-03T17:45:31.456943+0000","last_became_active":"2023-10-03T17:34:42.272306+0000","last_became_peered":"2023-10-03T17:34:42.272306+0000","last_unstale":"2023-10-03T17:45:31.456943+0000","last_undegraded":"2023-10-03T17:45:31.456943+0000","last_fullsized":"2023-10-03T17:45:31.456943+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'20","last_scrub_stamp":"2023-10-03T17:40:10.708060+0000","last_deep_scrub":"46'20","last_deep_scrub_stamp":"2023-10-03T17:40:10.708060+0000","last_clean_scrub_stamp":"2023-10-03T17:40:10.708060+0000","objects_scrubbed":7,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:03:09.097252+0000","scrub_duration":0.042279491000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":704,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":1116,"num_read_kb":903,"num_write":45,"num_write_kb":41,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":301,"num_omap_keys":10,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"61'75476","reported_seq":135064,"reported_epoch":61,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T17:45:40.415966+0000","last_change":"2023-10-03T17:40:29.562114+0000","last_active":"2023-10-03T17:45:40.415966+0000","last_peered":"2023-10-03T17:45:40.415966+0000","last_clean":"2023-10-03T17:45:40.415966+0000","last_became_active":"2023-10-03T17:34:48.310612+0000","last_became_peered":"2023-10-03T17:34:48.310612+0000","last_unstale":"2023-10-03T17:45:40.415966+0000","last_undegraded":"2023-10-03T17:45:40.415966+0000","last_fullsized":"2023-10-03T17:45:40.415966+0000","mapping_epoch":16,"log_start":"55'65381","ondisk_log_start":"55'65381","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_clean_scrub_stamp":"2023-10-03T17:34:47.281095+0000","objects_scrubbed":3545,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":20,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 310s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022375120000000001,"stat_sum":{"num_bytes":5405134867,"num_objects":7555,"num_object_clones":6267,"num_object_copies":15110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7555,"num_whiteouts":0,"num_read":65474,"num_read_kb":244136,"num_write":67004,"num_write_kb":268010,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"59'24","reported_seq":112,"reported_epoch":61,"state":"active+clean","last_fresh":"2023-10-03T17:45:06.351443+0000","last_change":"2023-10-03T17:40:24.472734+0000","last_active":"2023-10-03T17:45:06.351443+0000","last_peered":"2023-10-03T17:45:06.351443+0000","last_clean":"2023-10-03T17:45:06.351443+0000","last_became_active":"2023-10-03T17:34:42.270194+0000","last_became_peered":"2023-10-03T17:34:42.270194+0000","last_unstale":"2023-10-03T17:45:06.351443+0000","last_undegraded":"2023-10-03T17:45:06.351443+0000","last_fullsized":"2023-10-03T17:45:06.351443+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'8","last_scrub_stamp":"2023-10-03T17:40:24.472573+0000","last_deep_scrub":"46'8","last_deep_scrub_stamp":"2023-10-03T17:40:24.472573+0000","last_clean_scrub_stamp":"2023-10-03T17:40:24.472573+0000","objects_scrubbed":4,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T18:28:04.735033+0000","scrub_duration":0.024990637999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":636,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":29,"num_read_kb":11,"num_write":20,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"46'99","reported_seq":203,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-10-03T17:45:05.340127+0000","last_change":"2023-10-03T17:40:10.452341+0000","last_active":"2023-10-03T17:45:05.340127+0000","last_peered":"2023-10-03T17:45:05.340127+0000","last_clean":"2023-10-03T17:45:05.340127+0000","last_became_active":"2023-10-03T17:34:39.502393+0000","last_became_peered":"2023-10-03T17:34:39.502393+0000","last_unstale":"2023-10-03T17:45:05.340127+0000","last_undegraded":"2023-10-03T17:45:05.340127+0000","last_fullsized":"2023-10-03T17:45:05.340127+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'99","last_scrub_stamp":"2023-10-03T17:40:10.449099+0000","last_deep_scrub":"46'99","last_deep_scrub_stamp":"2023-10-03T17:40:10.449099+0000","last_clean_scrub_stamp":"2023-10-03T17:40:10.449099+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T20:18:04.958666+0000","scrub_duration":0.043225012,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":21503062035,"num_objects":29229,"num_object_clones":24108,"num_object_copies":58458,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29229,"num_whiteouts":0,"num_read":261399,"num_read_kb":974264,"num_write":265851,"num_write_kb":1063398,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2177536000,"data_stored":10071498790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40176,"ondisk_log_size":40176,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":5372,"num_objects":143,"num_object_clones":0,"num_object_copies":286,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":143,"num_whiteouts":0,"num_read":23527,"num_read_kb":19618,"num_write":743,"num_write_kb":683,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":983040,"data_stored":10744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7851,"ondisk_log_size":7851,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":7460,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738507,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2359384,"kb_used_data":1063184,"kb_used_omap":1,"kb_used_meta":1296190,"kb_avail":92012456,"statfs":{"total":96636764160,"available":94220754944,"internally_reserved":0,"allocated":1088700416,"data_stored":5035781482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1327299014},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 17:44:55 2023","interfaces":[{"interface":"back","average":{"1min":0.25800000000000001,"5min":0.40500000000000003,"15min":0.45700000000000002},"min":{"1min":0.16500000000000001,"5min":0.158,"15min":0.14299999999999999},"max":{"1min":0.45600000000000002,"5min":4.1539999999999999,"15min":4.4640000000000004},"last":0.29899999999999999},{"interface":"front","average":{"1min":0.30299999999999999,"5min":0.44,"15min":0.47099999999999997},"min":{"1min":0.20999999999999999,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":0.54800000000000004,"5min":4.4180000000000001,"15min":4.4180000000000001},"last":0.20899999999999999}]},{"osd":1,"last update":"Tue Oct 3 17:44:55 2023","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.55100000000000005,"15min":0.52500000000000002},"min":{"1min":0.19800000000000001,"5min":0.17599999999999999,"15min":0.16600000000000001},"max":{"1min":8.1059999999999999,"5min":8.1059999999999999,"15min":8.1059999999999999},"last":0.184},{"interface":"front","average":{"1min":0.34599999999999997,"5min":0.42999999999999999,"15min":0.48299999999999998},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.64400000000000002,"5min":3.9740000000000002,"15min":3.9740000000000002},"last":0.248}]}]},{"osd":1,"up_from":8,"seq":34359738505,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1219964,"kb_used_data":525424,"kb_used_omap":1,"kb_used_meta":694526,"kb_avail":93151876,"statfs":{"total":96636764160,"available":95387521024,"internally_reserved":0,"allocated":538034176,"data_stored":2561355174,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":711195078},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,5,11,2,1,9,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 17:45:08 2023","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.32900000000000001,"15min":0.41399999999999998},"min":{"1min":0.17199999999999999,"5min":0.17100000000000001,"15min":0.161},"max":{"1min":2.3809999999999998,"5min":2.3809999999999998,"15min":4.1879999999999997},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.27500000000000002,"5min":0.41399999999999998,"15min":0.44600000000000001},"min":{"1min":0.17799999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.46200000000000002,"5min":4.5759999999999996,"15min":4.5759999999999996},"last":0.29699999999999999}]},{"osd":2,"last update":"Tue Oct 3 17:45:08 2023","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.39000000000000001,"15min":0.44},"min":{"1min":0.214,"5min":0.187,"15min":0.159},"max":{"1min":1.3100000000000001,"5min":3.0390000000000001,"15min":3.0390000000000001},"last":0.251},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.35299999999999998,"15min":0.443},"min":{"1min":0.16400000000000001,"5min":0.16400000000000001,"15min":0.16400000000000001},"max":{"1min":1.337,"5min":3.0699999999999998,"15min":3.0699999999999998},"last":0.33500000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738505,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1223628,"kb_used_data":538820,"kb_used_omap":1,"kb_used_meta":684798,"kb_avail":93148212,"statfs":{"total":96636764160,"available":95383769088,"internally_reserved":0,"allocated":551751680,"data_stored":2475144883,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":701233606},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,4,3,6,12],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 17:44:52 2023","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.45100000000000001,"15min":0.42699999999999999},"min":{"1min":0.185,"5min":0.154,"15min":0.154},"max":{"1min":1.337,"5min":5.7649999999999997,"15min":5.7649999999999997},"last":0.30299999999999999},{"interface":"front","average":{"1min":0.311,"5min":0.51700000000000002,"15min":0.46999999999999997},"min":{"1min":0.191,"5min":0.19,"15min":0.17699999999999999},"max":{"1min":0.59099999999999997,"5min":9.4770000000000003,"15min":9.4770000000000003},"last":0.28499999999999998}]},{"osd":2,"last update":"Tue Oct 3 17:44:52 2023","interfaces":[{"interface":"back","average":{"1min":0.32400000000000001,"5min":0.53200000000000003,"15min":0.52600000000000002},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.44900000000000001,"5min":9.4149999999999991,"15min":9.4149999999999991},"last":3.9769999999999999},{"interface":"front","average":{"1min":0.434,"5min":0.42299999999999999,"15min":0.48399999999999999},"min":{"1min":0.19700000000000001,"5min":0.189,"15min":0.189},"max":{"1min":1.9359999999999999,"5min":3.8100000000000001,"15min":3.8100000000000001},"last":3.9260000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":3612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":5372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":159744,"data_stored":1760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":551260160,"data_stored":2473181184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":537690112,"data_stored":2563170323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1088585728,"data_stored":5035147283,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-03T17:45:42.168 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-03T17:34:47.281095+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=34, tm_sec=47, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=40, tm_sec=6, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T17:45:42.168 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-03T17:34:47.281095+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=34, tm_sec=47, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=40, tm_sec=6, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T17:45:42.169 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-03T17:34:47.281095+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=34, tm_sec=47, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=40, tm_sec=6, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T17:45:42.169 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.3 2023-10-03T17:45:42.751 INFO:teuthology.orchestra.run.smithi111.stderr:instructing pg 3.3 on osd.0 to deep-scrub 2023-10-03T17:45:42.782 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.0 2023-10-03T17:45:43.306 INFO:teuthology.orchestra.run.smithi111.stderr:instructing pg 3.0 on osd.1 to deep-scrub 2023-10-03T17:45:43.339 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.2 2023-10-03T17:45:43.862 INFO:teuthology.orchestra.run.smithi111.stderr:instructing pg 3.2 on osd.1 to deep-scrub 2023-10-03T17:45:43.897 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T17:46:03.898 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T17:46:04.395 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-03T17:46:04.396 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-10-03T17:46:04.421 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":407,"stamp":"2023-10-03T17:46:04.107436+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":21542782718,"num_objects":28878,"num_object_clones":23613,"num_object_copies":57756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28878,"num_whiteouts":0,"num_read":293234,"num_read_kb":1022644,"num_write":273812,"num_write_kb":1094119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":48292,"ondisk_log_size":48292,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":4951572,"kb_used_data":2183548,"kb_used_omap":4,"kb_used_meta":2767995,"kb_avail":278163948,"statfs":{"total":289910292480,"available":284839882752,"internally_reserved":0,"allocated":2235953152,"data_stored":9960223171,"data_compressed":7460,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4782,"internal_metadata":2834427218},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,3,0,6,9,17,17,10],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":9707471,"num_objects":-180,"num_object_clones":-179,"num_object_copies":-360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-180,"num_whiteouts":0,"num_read":2843,"num_read_kb":7635,"num_write":1818,"num_write_kb":7092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000591"},"pg_stats":[{"pgid":"2.7","version":"62'1461","reported_seq":2808,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-10-03T17:46:00.387916+0000","last_change":"2023-10-03T17:40:17.407011+0000","last_active":"2023-10-03T17:46:00.387916+0000","last_peered":"2023-10-03T17:46:00.387916+0000","last_clean":"2023-10-03T17:46:00.387916+0000","last_became_active":"2023-10-03T17:34:42.276624+0000","last_became_peered":"2023-10-03T17:34:42.276624+0000","last_unstale":"2023-10-03T17:46:00.387916+0000","last_undegraded":"2023-10-03T17:46:00.387916+0000","last_fullsized":"2023-10-03T17:46:00.387916+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'673","last_scrub_stamp":"2023-10-03T17:40:17.406813+0000","last_deep_scrub":"46'673","last_deep_scrub_stamp":"2023-10-03T17:40:17.406813+0000","last_clean_scrub_stamp":"2023-10-03T17:40:17.406813+0000","objects_scrubbed":7,"log_size":1461,"log_dups_size":0,"ondisk_log_size":1461,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T21:38:35.056889+0000","scrub_duration":0.013778076,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":3526,"num_read_kb":2900,"num_write":112,"num_write_kb":100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":903,"num_omap_keys":30,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"62'2438","reported_seq":5481,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-10-03T17:46:00.394509+0000","last_change":"2023-10-03T17:40:13.376519+0000","last_active":"2023-10-03T17:46:00.394509+0000","last_peered":"2023-10-03T17:46:00.394509+0000","last_clean":"2023-10-03T17:46:00.394509+0000","last_became_active":"2023-10-03T17:34:42.271201+0000","last_became_peered":"2023-10-03T17:34:42.271201+0000","last_unstale":"2023-10-03T17:46:00.394509+0000","last_undegraded":"2023-10-03T17:46:00.394509+0000","last_fullsized":"2023-10-03T17:46:00.394509+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'1129","last_scrub_stamp":"2023-10-03T17:40:13.376332+0000","last_deep_scrub":"46'1129","last_deep_scrub_stamp":"2023-10-03T17:40:13.376332+0000","last_clean_scrub_stamp":"2023-10-03T17:40:13.376332+0000","objects_scrubbed":11,"log_size":2438,"log_dups_size":0,"ondisk_log_size":2438,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T18:59:17.088585+0000","scrub_duration":0.015998675,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":878,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":6965,"num_read_kb":5967,"num_write":208,"num_write_kb":193,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2912,"num_omap_keys":93,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"62'32","reported_seq":127,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-10-03T17:46:00.388984+0000","last_change":"2023-10-03T17:40:25.496010+0000","last_active":"2023-10-03T17:46:00.388984+0000","last_peered":"2023-10-03T17:46:00.388984+0000","last_clean":"2023-10-03T17:46:00.388984+0000","last_became_active":"2023-10-03T17:34:42.276941+0000","last_became_peered":"2023-10-03T17:34:42.276941+0000","last_unstale":"2023-10-03T17:46:00.388984+0000","last_undegraded":"2023-10-03T17:46:00.388984+0000","last_fullsized":"2023-10-03T17:46:00.388984+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'14","last_scrub_stamp":"2023-10-03T17:40:25.495827+0000","last_deep_scrub":"46'14","last_deep_scrub_stamp":"2023-10-03T17:40:25.495827+0000","last_clean_scrub_stamp":"2023-10-03T17:40:25.495827+0000","objects_scrubbed":7,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:05:44.939663+0000","scrub_duration":0.0133078,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":956,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":41,"num_read_kb":14,"num_write":25,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"62'1952","reported_seq":4179,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-10-03T17:46:00.394600+0000","last_change":"2023-10-03T17:40:26.513902+0000","last_active":"2023-10-03T17:46:00.394600+0000","last_peered":"2023-10-03T17:46:00.394600+0000","last_clean":"2023-10-03T17:46:00.394600+0000","last_became_active":"2023-10-03T17:34:42.276459+0000","last_became_peered":"2023-10-03T17:34:42.276459+0000","last_unstale":"2023-10-03T17:46:00.394600+0000","last_undegraded":"2023-10-03T17:46:00.394600+0000","last_fullsized":"2023-10-03T17:46:00.394600+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'964","last_scrub_stamp":"2023-10-03T17:40:26.513729+0000","last_deep_scrub":"46'964","last_deep_scrub_stamp":"2023-10-03T17:40:26.513729+0000","last_clean_scrub_stamp":"2023-10-03T17:40:26.513729+0000","objects_scrubbed":10,"log_size":1952,"log_dups_size":0,"ondisk_log_size":1952,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:41:33.182751+0000","scrub_duration":0.018190280999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":333,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":5119,"num_read_kb":4271,"num_write":148,"num_write_kb":137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1731,"num_omap_keys":80,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"62'78964","reported_seq":141962,"reported_epoch":62,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T17:46:00.382019+0000","last_change":"2023-10-03T17:40:11.716583+0000","last_active":"2023-10-03T17:46:00.382019+0000","last_peered":"2023-10-03T17:46:00.382019+0000","last_clean":"2023-10-03T17:46:00.382019+0000","last_became_active":"2023-10-03T17:34:48.310701+0000","last_became_peered":"2023-10-03T17:34:48.310701+0000","last_unstale":"2023-10-03T17:46:00.382019+0000","last_undegraded":"2023-10-03T17:46:00.382019+0000","last_fullsized":"2023-10-03T17:46:00.382019+0000","mapping_epoch":16,"log_start":"55'68876","ondisk_log_start":"55'68876","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_clean_scrub_stamp":"2023-10-03T17:34:47.281095+0000","objects_scrubbed":3305,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":20,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 348s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040768800000000002,"stat_sum":{"num_bytes":5552967680,"num_objects":7829,"num_object_clones":6509,"num_object_copies":15658,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7829,"num_whiteouts":0,"num_read":69107,"num_read_kb":257800,"num_write":70515,"num_write_kb":282060,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"62'1930","reported_seq":3668,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-10-03T17:46:00.321068+0000","last_change":"2023-10-03T17:40:08.675119+0000","last_active":"2023-10-03T17:46:00.321068+0000","last_peered":"2023-10-03T17:46:00.321068+0000","last_clean":"2023-10-03T17:46:00.321068+0000","last_became_active":"2023-10-03T17:34:42.270625+0000","last_became_peered":"2023-10-03T17:34:42.270625+0000","last_unstale":"2023-10-03T17:46:00.321068+0000","last_undegraded":"2023-10-03T17:46:00.321068+0000","last_fullsized":"2023-10-03T17:46:00.321068+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'841","last_scrub_stamp":"2023-10-03T17:40:08.674910+0000","last_deep_scrub":"46'841","last_deep_scrub_stamp":"2023-10-03T17:40:08.674910+0000","last_clean_scrub_stamp":"2023-10-03T17:40:08.674910+0000","objects_scrubbed":11,"log_size":1930,"log_dups_size":0,"ondisk_log_size":1930,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T21:05:42.790614+0000","scrub_duration":0.039189555000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":808,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":4493,"num_read_kb":3709,"num_write":148,"num_write_kb":137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2564,"num_omap_keys":60,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"62'73965","reported_seq":133365,"reported_epoch":62,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T17:46:00.667596+0000","last_change":"2023-10-03T17:43:28.494717+0000","last_active":"2023-10-03T17:46:00.667596+0000","last_peered":"2023-10-03T17:46:00.667596+0000","last_clean":"2023-10-03T17:46:00.667596+0000","last_became_active":"2023-10-03T17:34:48.310555+0000","last_became_peered":"2023-10-03T17:34:48.310555+0000","last_unstale":"2023-10-03T17:46:00.667596+0000","last_undegraded":"2023-10-03T17:46:00.667596+0000","last_fullsized":"2023-10-03T17:46:00.667596+0000","mapping_epoch":16,"log_start":"55'63960","ondisk_log_start":"55'63960","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_clean_scrub_stamp":"2023-10-03T17:34:47.281095+0000","objects_scrubbed":2326,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":20,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 152s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086289699999999995,"stat_sum":{"num_bytes":5227151360,"num_objects":7381,"num_object_clones":6139,"num_object_copies":14762,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7381,"num_whiteouts":0,"num_read":65241,"num_read_kb":242796,"num_write":65959,"num_write_kb":263836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"61'113","reported_seq":1400,"reported_epoch":61,"state":"active+clean","last_fresh":"2023-10-03T17:46:00.103954+0000","last_change":"2023-10-03T17:40:17.641493+0000","last_active":"2023-10-03T17:46:00.103954+0000","last_peered":"2023-10-03T17:46:00.103954+0000","last_clean":"2023-10-03T17:46:00.103954+0000","last_became_active":"2023-10-03T17:34:42.278196+0000","last_became_peered":"2023-10-03T17:34:42.278196+0000","last_unstale":"2023-10-03T17:46:00.103954+0000","last_undegraded":"2023-10-03T17:46:00.103954+0000","last_fullsized":"2023-10-03T17:46:00.103954+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'30","last_scrub_stamp":"2023-10-03T17:40:17.640969+0000","last_deep_scrub":"46'30","last_deep_scrub_stamp":"2023-10-03T17:40:17.640969+0000","last_clean_scrub_stamp":"2023-10-03T17:40:17.640969+0000","objects_scrubbed":6,"log_size":113,"log_dups_size":0,"ondisk_log_size":113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:04:02.954896+0000","scrub_duration":0.071054565,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":420,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":3298,"num_read_kb":2719,"num_write":86,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":903,"num_omap_keys":30,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"61'77542","reported_seq":137956,"reported_epoch":62,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T17:46:00.333718+0000","last_change":"2023-10-03T17:45:57.732189+0000","last_active":"2023-10-03T17:46:00.333718+0000","last_peered":"2023-10-03T17:46:00.333718+0000","last_clean":"2023-10-03T17:46:00.333718+0000","last_became_active":"2023-10-03T17:34:48.310912+0000","last_became_peered":"2023-10-03T17:34:48.310912+0000","last_unstale":"2023-10-03T17:46:00.333718+0000","last_undegraded":"2023-10-03T17:46:00.333718+0000","last_fullsized":"2023-10-03T17:46:00.333718+0000","mapping_epoch":16,"log_start":"55'67492","ondisk_log_start":"55'67492","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'58196","last_scrub_stamp":"2023-10-03T17:43:26.193075+0000","last_deep_scrub":"53'58196","last_deep_scrub_stamp":"2023-10-03T17:43:26.193075+0000","last_clean_scrub_stamp":"2023-10-03T17:43:26.193075+0000","objects_scrubbed":2326,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":196,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T20:19:15.400687+0000","scrub_duration":195.44739730800001,"objects_trimmed":0,"snaptrim_duration":51.370317178000001,"stat_sum":{"num_bytes":5348499456,"num_objects":6290,"num_object_clones":5019,"num_object_copies":12580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6290,"num_whiteouts":0,"num_read":66907,"num_read_kb":250448,"num_write":67747,"num_write_kb":270988,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"54","length":"13"}]},{"pgid":"2.0","version":"61'59","reported_seq":560,"reported_epoch":61,"state":"active+clean","last_fresh":"2023-10-03T17:46:00.091142+0000","last_change":"2023-10-03T17:40:10.708344+0000","last_active":"2023-10-03T17:46:00.091142+0000","last_peered":"2023-10-03T17:46:00.091142+0000","last_clean":"2023-10-03T17:46:00.091142+0000","last_became_active":"2023-10-03T17:34:42.272306+0000","last_became_peered":"2023-10-03T17:34:42.272306+0000","last_unstale":"2023-10-03T17:46:00.091142+0000","last_undegraded":"2023-10-03T17:46:00.091142+0000","last_fullsized":"2023-10-03T17:46:00.091142+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'20","last_scrub_stamp":"2023-10-03T17:40:10.708060+0000","last_deep_scrub":"46'20","last_deep_scrub_stamp":"2023-10-03T17:40:10.708060+0000","last_clean_scrub_stamp":"2023-10-03T17:40:10.708060+0000","objects_scrubbed":7,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:03:09.097252+0000","scrub_duration":0.042279491000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":704,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":1175,"num_read_kb":950,"num_write":45,"num_write_kb":41,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":301,"num_omap_keys":10,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"62'78126","reported_seq":139194,"reported_epoch":62,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T17:46:00.659514+0000","last_change":"2023-10-03T17:46:00.480286+0000","last_active":"2023-10-03T17:46:00.659514+0000","last_peered":"2023-10-03T17:46:00.659514+0000","last_clean":"2023-10-03T17:46:00.659514+0000","last_became_active":"2023-10-03T17:34:48.310612+0000","last_became_peered":"2023-10-03T17:34:48.310612+0000","last_unstale":"2023-10-03T17:46:00.659514+0000","last_undegraded":"2023-10-03T17:46:00.659514+0000","last_fullsized":"2023-10-03T17:46:00.659514+0000","mapping_epoch":16,"log_start":"55'68093","ondisk_log_start":"55'68093","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'75807","last_scrub_stamp":"2023-10-03T17:45:43.833310+0000","last_deep_scrub":"61'75807","last_deep_scrub_stamp":"2023-10-03T17:45:43.833310+0000","last_clean_scrub_stamp":"2023-10-03T17:45:43.833310+0000","objects_scrubbed":3607,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":13,"last_scrub_duration":315,"scrub_schedule":"queued for deep scrub","scrub_duration":314.26374841099999,"objects_trimmed":13,"snaptrim_duration":0.174418343,"stat_sum":{"num_bytes":5413568531,"num_objects":7234,"num_object_clones":5946,"num_object_copies":14468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7234,"num_whiteouts":0,"num_read":67223,"num_read_kb":250964,"num_write":68654,"num_write_kb":274610,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"54","length":"7"}]},{"pgid":"2.3","version":"62'32","reported_seq":125,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-10-03T17:46:00.458067+0000","last_change":"2023-10-03T17:40:24.472734+0000","last_active":"2023-10-03T17:46:00.458067+0000","last_peered":"2023-10-03T17:46:00.458067+0000","last_clean":"2023-10-03T17:46:00.458067+0000","last_became_active":"2023-10-03T17:34:42.270194+0000","last_became_peered":"2023-10-03T17:34:42.270194+0000","last_unstale":"2023-10-03T17:46:00.458067+0000","last_undegraded":"2023-10-03T17:46:00.458067+0000","last_fullsized":"2023-10-03T17:46:00.458067+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'8","last_scrub_stamp":"2023-10-03T17:40:24.472573+0000","last_deep_scrub":"46'8","last_deep_scrub_stamp":"2023-10-03T17:40:24.472573+0000","last_clean_scrub_stamp":"2023-10-03T17:40:24.472573+0000","objects_scrubbed":4,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T18:28:04.735033+0000","scrub_duration":0.024990637999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":636,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":33,"num_read_kb":15,"num_write":28,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"46'99","reported_seq":204,"reported_epoch":61,"state":"active+clean","last_fresh":"2023-10-03T17:45:06.353782+0000","last_change":"2023-10-03T17:40:10.452341+0000","last_active":"2023-10-03T17:45:06.353782+0000","last_peered":"2023-10-03T17:45:06.353782+0000","last_clean":"2023-10-03T17:45:06.353782+0000","last_became_active":"2023-10-03T17:34:39.502393+0000","last_became_peered":"2023-10-03T17:34:39.502393+0000","last_unstale":"2023-10-03T17:45:06.353782+0000","last_undegraded":"2023-10-03T17:45:06.353782+0000","last_fullsized":"2023-10-03T17:45:06.353782+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'99","last_scrub_stamp":"2023-10-03T17:40:10.449099+0000","last_deep_scrub":"46'99","last_deep_scrub_stamp":"2023-10-03T17:40:10.449099+0000","last_clean_scrub_stamp":"2023-10-03T17:40:10.449099+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T20:18:04.958666+0000","scrub_duration":0.043225012,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":21542187027,"num_objects":28734,"num_object_clones":23613,"num_object_copies":57468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28734,"num_whiteouts":0,"num_read":268478,"num_read_kb":1002008,"num_write":272875,"num_write_kb":1091494,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2235047936,"data_stored":9956704294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40176,"ondisk_log_size":40176,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":5323,"num_objects":142,"num_object_clones":0,"num_object_copies":284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":142,"num_whiteouts":0,"num_read":24650,"num_read_kb":20545,"num_write":800,"num_write_kb":731,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":10845,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8017,"ondisk_log_size":8017,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":7460,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738511,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2437384,"kb_used_data":1091200,"kb_used_omap":1,"kb_used_meta":1346174,"kb_avail":91934456,"statfs":{"total":96636764160,"available":94140882944,"internally_reserved":0,"allocated":1117388800,"data_stored":4979158378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1378482630},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,1,7,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 17:45:55 2023","interfaces":[{"interface":"back","average":{"1min":0.30099999999999999,"5min":0.40899999999999997,"15min":0.442},"min":{"1min":0.17999999999999999,"5min":0.158,"15min":0.14299999999999999},"max":{"1min":0.51100000000000001,"5min":4.1539999999999999,"15min":4.4640000000000004},"last":0.27600000000000002},{"interface":"front","average":{"1min":0.312,"5min":0.42699999999999999,"15min":0.45500000000000002},"min":{"1min":0.19700000000000001,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":0.54000000000000004,"5min":4.4180000000000001,"15min":4.4180000000000001},"last":0.32200000000000001}]},{"osd":1,"last update":"Tue Oct 3 17:45:55 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.61299999999999999,"15min":0.52700000000000002},"min":{"1min":0.184,"5min":0.17599999999999999,"15min":0.16600000000000001},"max":{"1min":4.0519999999999996,"5min":8.1059999999999999,"15min":8.1059999999999999},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.76100000000000001,"5min":0.501,"15min":0.49399999999999999},"min":{"1min":0.248,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":4.1120000000000001,"5min":4.1120000000000001,"15min":4.1120000000000001},"last":1.627}]}]},{"osd":1,"up_from":8,"seq":34359738509,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1251628,"kb_used_data":539044,"kb_used_omap":1,"kb_used_meta":712574,"kb_avail":93120212,"statfs":{"total":96636764160,"available":95355097088,"internally_reserved":0,"allocated":551981056,"data_stored":2537430438,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":729676230},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,1,4,5,15,10],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 17:45:08 2023","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.32900000000000001,"15min":0.41399999999999998},"min":{"1min":0.17199999999999999,"5min":0.17100000000000001,"15min":0.161},"max":{"1min":2.3809999999999998,"5min":2.3809999999999998,"15min":4.1879999999999997},"last":0.19900000000000001},{"interface":"front","average":{"1min":0.27500000000000002,"5min":0.41399999999999998,"15min":0.44600000000000001},"min":{"1min":0.17799999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.46200000000000002,"5min":4.5759999999999996,"15min":4.5759999999999996},"last":0.39300000000000002}]},{"osd":2,"last update":"Tue Oct 3 17:45:08 2023","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.39000000000000001,"15min":0.44},"min":{"1min":0.214,"5min":0.187,"15min":0.159},"max":{"1min":1.3100000000000001,"5min":3.0390000000000001,"15min":3.0390000000000001},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.35299999999999998,"15min":0.443},"min":{"1min":0.16400000000000001,"5min":0.16400000000000001,"15min":0.16400000000000001},"max":{"1min":1.337,"5min":3.0699999999999998,"15min":3.0699999999999998},"last":0.26900000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738509,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1262560,"kb_used_data":553304,"kb_used_omap":1,"kb_used_meta":709246,"kb_avail":93109280,"statfs":{"total":96636764160,"available":95343902720,"internally_reserved":0,"allocated":566583296,"data_stored":2443634355,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":726268358},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,1,4,5,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 17:45:55 2023","interfaces":[{"interface":"back","average":{"1min":0.371,"5min":0.42999999999999999,"15min":0.42199999999999999},"min":{"1min":0.219,"5min":0.16500000000000001,"15min":0.154},"max":{"1min":1.202,"5min":5.7649999999999997,"15min":5.7649999999999997},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.51400000000000001,"15min":0.46899999999999997},"min":{"1min":0.20699999999999999,"5min":0.19,"15min":0.17699999999999999},"max":{"1min":1.585,"5min":9.4770000000000003,"15min":9.4770000000000003},"last":0.40000000000000002}]},{"osd":2,"last update":"Tue Oct 3 17:45:55 2023","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.59999999999999998,"15min":0.53700000000000003},"min":{"1min":0.23200000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":4.1619999999999999,"5min":9.4149999999999991,"15min":9.4149999999999991},"last":0.47399999999999998},{"interface":"front","average":{"1min":1.01,"5min":0.54900000000000004,"15min":0.51400000000000001},"min":{"1min":0.22600000000000001,"5min":0.189,"15min":0.189},"max":{"1min":5.218,"5min":5.218,"15min":5.218},"last":0.35599999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":339968,"data_stored":3713,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":5372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":159744,"data_stored":1760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":566333440,"data_stored":2441969664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":551383040,"data_stored":2536595475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1117331456,"data_stored":4978139155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-03T17:46:04.423 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-03T17:34:47.281095+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=34, tm_sec=47, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=40, tm_sec=6, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T17:46:04.423 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-03T17:34:47.281095+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=34, tm_sec=47, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=40, tm_sec=6, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T17:46:04.424 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T17:46:13.512 INFO:tasks.ceph.osd.0.smithi111.stderr:2023-10-03T17:46:13.511+0000 7ffbac289640 -1 reset not still connected to 0x5563a7ff65b0 2023-10-03T17:46:13.512 INFO:tasks.ceph.osd.0.smithi111.stderr:2023-10-03T17:46:13.511+0000 7ffbac289640 -1 reset not still connected to 0x5563a7ff68f0 2023-10-03T17:46:13.513 INFO:tasks.ceph.osd.0.smithi111.stderr:2023-10-03T17:46:13.511+0000 7ffbac289640 -1 reset not still connected to 0x5563a838e1a0 2023-10-03T17:46:13.513 INFO:tasks.ceph.osd.0.smithi111.stderr:2023-10-03T17:46:13.511+0000 7ffbac289640 -1 reset not still connected to 0x5563a851aea0 2023-10-03T17:46:24.425 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T17:46:24.925 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-03T17:46:24.926 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-10-03T17:46:24.968 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":418,"stamp":"2023-10-03T17:46:24.110729+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":21575796527,"num_objects":31459,"num_object_clones":26193,"num_object_copies":62918,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31459,"num_whiteouts":0,"num_read":302279,"num_read_kb":1047732,"num_write":279700,"num_write_kb":1117131,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":48485,"ondisk_log_size":48485,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":5280276,"kb_used_data":2229496,"kb_used_omap":4,"kb_used_meta":3050747,"kb_avail":277835244,"statfs":{"total":289910292480,"available":284503289856,"internally_reserved":0,"allocated":2283003904,"data_stored":11133931218,"data_compressed":7460,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4782,"internal_metadata":3123965266},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,1,5,11,8,20,13,12],"upper_bound":512},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":16277504,"num_objects":912,"num_object_clones":912,"num_object_copies":1824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":912,"num_whiteouts":0,"num_read":4278,"num_read_kb":12255,"num_write":2819,"num_write_kb":11276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001982"},"pg_stats":[{"pgid":"2.7","version":"66'1500","reported_seq":2941,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-03T17:46:19.097829+0000","last_change":"2023-10-03T17:40:17.407011+0000","last_active":"2023-10-03T17:46:19.097829+0000","last_peered":"2023-10-03T17:46:19.097829+0000","last_clean":"2023-10-03T17:46:19.097829+0000","last_became_active":"2023-10-03T17:34:42.276624+0000","last_became_peered":"2023-10-03T17:34:42.276624+0000","last_unstale":"2023-10-03T17:46:19.097829+0000","last_undegraded":"2023-10-03T17:46:19.097829+0000","last_fullsized":"2023-10-03T17:46:19.097829+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'673","last_scrub_stamp":"2023-10-03T17:40:17.406813+0000","last_deep_scrub":"46'673","last_deep_scrub_stamp":"2023-10-03T17:40:17.406813+0000","last_clean_scrub_stamp":"2023-10-03T17:40:17.406813+0000","objects_scrubbed":7,"log_size":1500,"log_dups_size":0,"ondisk_log_size":1500,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T21:38:35.056889+0000","scrub_duration":0.013778076,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":3886,"num_read_kb":3202,"num_write":129,"num_write_kb":110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":903,"num_omap_keys":30,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"66'2511","reported_seq":5832,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-03T17:46:19.096826+0000","last_change":"2023-10-03T17:40:13.376519+0000","last_active":"2023-10-03T17:46:19.096826+0000","last_peered":"2023-10-03T17:46:19.096826+0000","last_clean":"2023-10-03T17:46:19.096826+0000","last_became_active":"2023-10-03T17:34:42.271201+0000","last_became_peered":"2023-10-03T17:34:42.271201+0000","last_unstale":"2023-10-03T17:46:19.096826+0000","last_undegraded":"2023-10-03T17:46:19.096826+0000","last_fullsized":"2023-10-03T17:46:19.096826+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'1129","last_scrub_stamp":"2023-10-03T17:40:13.376332+0000","last_deep_scrub":"46'1129","last_deep_scrub_stamp":"2023-10-03T17:40:13.376332+0000","last_clean_scrub_stamp":"2023-10-03T17:40:13.376332+0000","objects_scrubbed":11,"log_size":2511,"log_dups_size":0,"ondisk_log_size":2511,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T18:59:17.088585+0000","scrub_duration":0.015998675,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":674,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":7871,"num_read_kb":6742,"num_write":247,"num_write_kb":217,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2912,"num_omap_keys":93,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"64'33","reported_seq":132,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-10-03T17:46:03.321857+0000","last_change":"2023-10-03T17:40:25.496010+0000","last_active":"2023-10-03T17:46:03.321857+0000","last_peered":"2023-10-03T17:46:03.321857+0000","last_clean":"2023-10-03T17:46:03.321857+0000","last_became_active":"2023-10-03T17:34:42.276941+0000","last_became_peered":"2023-10-03T17:34:42.276941+0000","last_unstale":"2023-10-03T17:46:03.321857+0000","last_undegraded":"2023-10-03T17:46:03.321857+0000","last_fullsized":"2023-10-03T17:46:03.321857+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'14","last_scrub_stamp":"2023-10-03T17:40:25.495827+0000","last_deep_scrub":"46'14","last_deep_scrub_stamp":"2023-10-03T17:40:25.495827+0000","last_clean_scrub_stamp":"2023-10-03T17:40:25.495827+0000","objects_scrubbed":7,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:05:44.939663+0000","scrub_duration":0.0133078,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1007,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":41,"num_read_kb":14,"num_write":26,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"66'2005","reported_seq":4416,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-03T17:46:19.096500+0000","last_change":"2023-10-03T17:40:26.513902+0000","last_active":"2023-10-03T17:46:19.096500+0000","last_peered":"2023-10-03T17:46:19.096500+0000","last_clean":"2023-10-03T17:46:19.096500+0000","last_became_active":"2023-10-03T17:34:42.276459+0000","last_became_peered":"2023-10-03T17:34:42.276459+0000","last_unstale":"2023-10-03T17:46:19.096500+0000","last_undegraded":"2023-10-03T17:46:19.096500+0000","last_fullsized":"2023-10-03T17:46:19.096500+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'964","last_scrub_stamp":"2023-10-03T17:40:26.513729+0000","last_deep_scrub":"46'964","last_deep_scrub_stamp":"2023-10-03T17:40:26.513729+0000","last_clean_scrub_stamp":"2023-10-03T17:40:26.513729+0000","objects_scrubbed":10,"log_size":2005,"log_dups_size":0,"ondisk_log_size":2005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:41:33.182751+0000","scrub_duration":0.018190280999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":384,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":5729,"num_read_kb":4786,"num_write":173,"num_write_kb":154,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1731,"num_omap_keys":80,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"66'81607","reported_seq":145235,"reported_epoch":66,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T17:46:20.525863+0000","last_change":"2023-10-03T17:46:18.018943+0000","last_active":"2023-10-03T17:46:20.525863+0000","last_peered":"2023-10-03T17:46:20.525863+0000","last_clean":"2023-10-03T17:46:20.525863+0000","last_became_active":"2023-10-03T17:34:48.310701+0000","last_became_peered":"2023-10-03T17:34:48.310701+0000","last_unstale":"2023-10-03T17:46:20.525863+0000","last_undegraded":"2023-10-03T17:46:20.525863+0000","last_fullsized":"2023-10-03T17:46:20.525863+0000","mapping_epoch":16,"log_start":"55'71584","ondisk_log_start":"55'71584","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'80695","last_scrub_stamp":"2023-10-03T17:46:13.982571+0000","last_deep_scrub":"66'80695","last_deep_scrub_stamp":"2023-10-03T17:46:13.982571+0000","last_clean_scrub_stamp":"2023-10-03T17:46:13.982571+0000","objects_scrubbed":3410,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":39,"last_scrub_duration":363,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:49:51.684926+0000","scrub_duration":362.26574615599998,"objects_trimmed":98,"snaptrim_duration":12.68730364,"stat_sum":{"num_bytes":5561462784,"num_objects":8590,"num_object_clones":7270,"num_object_copies":17180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8590,"num_whiteouts":0,"num_read":70630,"num_read_kb":263700,"num_write":72013,"num_write_kb":288052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"54","length":"1"}]},{"pgid":"2.2","version":"66'2003","reported_seq":3918,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-03T17:46:17.527163+0000","last_change":"2023-10-03T17:40:08.675119+0000","last_active":"2023-10-03T17:46:17.527163+0000","last_peered":"2023-10-03T17:46:17.527163+0000","last_clean":"2023-10-03T17:46:17.527163+0000","last_became_active":"2023-10-03T17:34:42.270625+0000","last_became_peered":"2023-10-03T17:34:42.270625+0000","last_unstale":"2023-10-03T17:46:17.527163+0000","last_undegraded":"2023-10-03T17:46:17.527163+0000","last_fullsized":"2023-10-03T17:46:17.527163+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'841","last_scrub_stamp":"2023-10-03T17:40:08.674910+0000","last_deep_scrub":"46'841","last_deep_scrub_stamp":"2023-10-03T17:40:08.674910+0000","last_clean_scrub_stamp":"2023-10-03T17:40:08.674910+0000","objects_scrubbed":11,"log_size":2003,"log_dups_size":0,"ondisk_log_size":2003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T21:05:42.790614+0000","scrub_duration":0.039189555000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":858,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":5201,"num_read_kb":4296,"num_write":185,"num_write_kb":164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2564,"num_omap_keys":60,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"66'76370","reported_seq":136361,"reported_epoch":66,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T17:46:20.793710+0000","last_change":"2023-10-03T17:43:28.494717+0000","last_active":"2023-10-03T17:46:20.793710+0000","last_peered":"2023-10-03T17:46:20.793710+0000","last_clean":"2023-10-03T17:46:20.793710+0000","last_became_active":"2023-10-03T17:34:48.310555+0000","last_became_peered":"2023-10-03T17:34:48.310555+0000","last_unstale":"2023-10-03T17:46:20.793710+0000","last_undegraded":"2023-10-03T17:46:20.793710+0000","last_fullsized":"2023-10-03T17:46:20.793710+0000","mapping_epoch":16,"log_start":"55'66366","ondisk_log_start":"55'66366","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_clean_scrub_stamp":"2023-10-03T17:34:47.281095+0000","objects_scrubbed":2633,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":40,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 172s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086289699999999995,"stat_sum":{"num_bytes":5235154944,"num_objects":8180,"num_object_clones":6938,"num_object_copies":16360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8180,"num_whiteouts":0,"num_read":66612,"num_read_kb":248116,"num_write":67308,"num_write_kb":269232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"66'133","reported_seq":1561,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-03T17:46:17.875474+0000","last_change":"2023-10-03T17:40:17.641493+0000","last_active":"2023-10-03T17:46:17.875474+0000","last_peered":"2023-10-03T17:46:17.875474+0000","last_clean":"2023-10-03T17:46:17.875474+0000","last_became_active":"2023-10-03T17:34:42.278196+0000","last_became_peered":"2023-10-03T17:34:42.278196+0000","last_unstale":"2023-10-03T17:46:17.875474+0000","last_undegraded":"2023-10-03T17:46:17.875474+0000","last_fullsized":"2023-10-03T17:46:17.875474+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'30","last_scrub_stamp":"2023-10-03T17:40:17.640969+0000","last_deep_scrub":"46'30","last_deep_scrub_stamp":"2023-10-03T17:40:17.640969+0000","last_clean_scrub_stamp":"2023-10-03T17:40:17.640969+0000","objects_scrubbed":6,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:04:02.954896+0000","scrub_duration":0.071054565,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":471,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":3867,"num_read_kb":3192,"num_write":116,"num_write_kb":102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":903,"num_omap_keys":30,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"66'80445","reported_seq":141352,"reported_epoch":66,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T17:46:20.496774+0000","last_change":"2023-10-03T17:46:17.155067+0000","last_active":"2023-10-03T17:46:20.496774+0000","last_peered":"2023-10-03T17:46:20.496774+0000","last_clean":"2023-10-03T17:46:20.496774+0000","last_became_active":"2023-10-03T17:34:48.310912+0000","last_became_peered":"2023-10-03T17:34:48.310912+0000","last_unstale":"2023-10-03T17:46:20.496774+0000","last_undegraded":"2023-10-03T17:46:20.496774+0000","last_fullsized":"2023-10-03T17:46:20.496774+0000","mapping_epoch":16,"log_start":"61'70416","ondisk_log_start":"61'70416","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'58196","last_scrub_stamp":"2023-10-03T17:43:26.193075+0000","last_deep_scrub":"53'58196","last_deep_scrub_stamp":"2023-10-03T17:43:26.193075+0000","last_clean_scrub_stamp":"2023-10-03T17:43:26.193075+0000","objects_scrubbed":2326,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":16,"last_scrub_duration":196,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T21:49:35.095231+0000","scrub_duration":195.44739730800001,"objects_trimmed":246,"snaptrim_duration":11.818037178999999,"stat_sum":{"num_bytes":5356478464,"num_objects":6792,"num_object_clones":5521,"num_object_copies":13584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6792,"num_whiteouts":0,"num_read":68366,"num_read_kb":256148,"num_write":69162,"num_write_kb":276648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"54","length":"18"}]},{"pgid":"2.0","version":"64'68","reported_seq":614,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-03T17:46:12.914724+0000","last_change":"2023-10-03T17:40:10.708344+0000","last_active":"2023-10-03T17:46:12.914724+0000","last_peered":"2023-10-03T17:46:12.914724+0000","last_clean":"2023-10-03T17:46:12.914724+0000","last_became_active":"2023-10-03T17:34:42.272306+0000","last_became_peered":"2023-10-03T17:34:42.272306+0000","last_unstale":"2023-10-03T17:46:12.914724+0000","last_undegraded":"2023-10-03T17:46:12.914724+0000","last_fullsized":"2023-10-03T17:46:12.914724+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'20","last_scrub_stamp":"2023-10-03T17:40:10.708060+0000","last_deep_scrub":"46'20","last_deep_scrub_stamp":"2023-10-03T17:40:10.708060+0000","last_clean_scrub_stamp":"2023-10-03T17:40:10.708060+0000","objects_scrubbed":7,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:03:09.097252+0000","scrub_duration":0.042279491000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":805,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":1326,"num_read_kb":1077,"num_write":58,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":301,"num_omap_keys":10,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"66'81051","reported_seq":142534,"reported_epoch":66,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T17:46:20.809383+0000","last_change":"2023-10-03T17:46:17.491180+0000","last_active":"2023-10-03T17:46:20.809383+0000","last_peered":"2023-10-03T17:46:20.809383+0000","last_clean":"2023-10-03T17:46:20.809383+0000","last_became_active":"2023-10-03T17:34:48.310612+0000","last_became_peered":"2023-10-03T17:34:48.310612+0000","last_unstale":"2023-10-03T17:46:20.809383+0000","last_undegraded":"2023-10-03T17:46:20.809383+0000","last_fullsized":"2023-10-03T17:46:20.809383+0000","mapping_epoch":16,"log_start":"58'71007","ondisk_log_start":"58'71007","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'75807","last_scrub_stamp":"2023-10-03T17:45:43.833310+0000","last_deep_scrub":"61'75807","last_deep_scrub_stamp":"2023-10-03T17:45:43.833310+0000","last_clean_scrub_stamp":"2023-10-03T17:45:43.833310+0000","objects_scrubbed":3607,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":29,"last_scrub_duration":315,"scrub_schedule":"queued for deep scrub","scrub_duration":314.26374841099999,"objects_trimmed":254,"snaptrim_duration":12.160941505,"stat_sum":{"num_bytes":5422104595,"num_objects":7752,"num_object_clones":6464,"num_object_copies":15504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7752,"num_whiteouts":0,"num_read":68610,"num_read_kb":256352,"num_write":70117,"num_write_kb":280462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"54","length":"b"}]},{"pgid":"2.3","version":"64'33","reported_seq":131,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-10-03T17:46:03.321895+0000","last_change":"2023-10-03T17:40:24.472734+0000","last_active":"2023-10-03T17:46:03.321895+0000","last_peered":"2023-10-03T17:46:03.321895+0000","last_clean":"2023-10-03T17:46:03.321895+0000","last_became_active":"2023-10-03T17:34:42.270194+0000","last_became_peered":"2023-10-03T17:34:42.270194+0000","last_unstale":"2023-10-03T17:46:03.321895+0000","last_undegraded":"2023-10-03T17:46:03.321895+0000","last_fullsized":"2023-10-03T17:46:03.321895+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'8","last_scrub_stamp":"2023-10-03T17:40:24.472573+0000","last_deep_scrub":"46'8","last_deep_scrub_stamp":"2023-10-03T17:40:24.472573+0000","last_clean_scrub_stamp":"2023-10-03T17:40:24.472573+0000","objects_scrubbed":4,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T18:28:04.735033+0000","scrub_duration":0.024990637999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":585,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":34,"num_read_kb":16,"num_write":29,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"46'99","reported_seq":208,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-10-03T17:46:03.322956+0000","last_change":"2023-10-03T17:40:10.452341+0000","last_active":"2023-10-03T17:46:03.322956+0000","last_peered":"2023-10-03T17:46:03.322956+0000","last_clean":"2023-10-03T17:46:03.322956+0000","last_became_active":"2023-10-03T17:34:39.502393+0000","last_became_peered":"2023-10-03T17:34:39.502393+0000","last_unstale":"2023-10-03T17:46:03.322956+0000","last_undegraded":"2023-10-03T17:46:03.322956+0000","last_fullsized":"2023-10-03T17:46:03.322956+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'99","last_scrub_stamp":"2023-10-03T17:40:10.449099+0000","last_deep_scrub":"46'99","last_deep_scrub_stamp":"2023-10-03T17:40:10.449099+0000","last_clean_scrub_stamp":"2023-10-03T17:40:10.449099+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T20:18:04.958666+0000","scrub_duration":0.043225012,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":21575200787,"num_objects":31314,"num_object_clones":26193,"num_object_copies":62628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31314,"num_whiteouts":0,"num_read":274218,"num_read_kb":1024316,"num_write":278600,"num_write_kb":1114394,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2281877504,"data_stored":11151822886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40100,"ondisk_log_size":40100,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":5372,"num_objects":143,"num_object_clones":0,"num_object_copies":286,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":143,"num_whiteouts":0,"num_read":27955,"num_read_kb":23325,"num_write":963,"num_write_kb":843,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":983040,"data_stored":10744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8286,"ondisk_log_size":8286,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":7460,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738515,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2587976,"kb_used_data":1114044,"kb_used_omap":1,"kb_used_meta":1473918,"kb_avail":91783864,"statfs":{"total":96636764160,"available":93986676736,"internally_reserved":0,"allocated":1140781056,"data_stored":5564009684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1509292486},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,2,4,5,10,4,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 17:45:55 2023","interfaces":[{"interface":"back","average":{"1min":0.30099999999999999,"5min":0.40899999999999997,"15min":0.442},"min":{"1min":0.17999999999999999,"5min":0.158,"15min":0.14299999999999999},"max":{"1min":0.51100000000000001,"5min":4.1539999999999999,"15min":4.4640000000000004},"last":0.28100000000000003},{"interface":"front","average":{"1min":0.312,"5min":0.42699999999999999,"15min":0.45500000000000002},"min":{"1min":0.19700000000000001,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":0.54000000000000004,"5min":4.4180000000000001,"15min":4.4180000000000001},"last":0.34899999999999998}]},{"osd":1,"last update":"Tue Oct 3 17:45:55 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.61299999999999999,"15min":0.52700000000000002},"min":{"1min":0.184,"5min":0.17599999999999999,"15min":0.16600000000000001},"max":{"1min":4.0519999999999996,"5min":8.1059999999999999,"15min":8.1059999999999999},"last":0.247},{"interface":"front","average":{"1min":0.76100000000000001,"5min":0.501,"15min":0.49399999999999999},"min":{"1min":0.248,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":4.1120000000000001,"5min":4.1120000000000001,"15min":4.1120000000000001},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738513,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1368624,"kb_used_data":550376,"kb_used_omap":1,"kb_used_meta":818238,"kb_avail":93003216,"statfs":{"total":96636764160,"available":95235293184,"internally_reserved":0,"allocated":563585024,"data_stored":2836237483,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":837876166},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,1,1,3,2,5,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 17:46:09 2023","interfaces":[{"interface":"back","average":{"1min":0.33200000000000002,"5min":0.33000000000000002,"15min":0.40100000000000002},"min":{"1min":0.19800000000000001,"5min":0.17199999999999999,"15min":0.161},"max":{"1min":0.88300000000000001,"5min":2.3809999999999998,"15min":4.1879999999999997},"last":0.307},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.45100000000000001,"15min":0.443},"min":{"1min":0.17699999999999999,"5min":0.17599999999999999,"15min":0.17399999999999999},"max":{"1min":2.4809999999999999,"5min":4.5759999999999996,"15min":4.5759999999999996},"last":0.29499999999999998}]},{"osd":2,"last update":"Tue Oct 3 17:46:09 2023","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.38,"15min":0.432},"min":{"1min":0.21199999999999999,"5min":0.187,"15min":0.159},"max":{"1min":1.026,"5min":2.6110000000000002,"15min":3.0390000000000001},"last":0.20200000000000001},{"interface":"front","average":{"1min":0.42099999999999999,"5min":0.34300000000000003,"15min":0.433},"min":{"1min":0.26800000000000002,"5min":0.16400000000000001,"15min":0.16400000000000001},"max":{"1min":0.79100000000000004,"5min":1.337,"15min":3.0699999999999998},"last":0.27100000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738513,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1323676,"kb_used_data":565076,"kb_used_omap":1,"kb_used_meta":758590,"kb_avail":93048164,"statfs":{"total":96636764160,"available":95281319936,"internally_reserved":0,"allocated":578637824,"data_stored":2733684051,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":776796614},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,6,0,8,4,6],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 17:45:55 2023","interfaces":[{"interface":"back","average":{"1min":0.371,"5min":0.42999999999999999,"15min":0.42199999999999999},"min":{"1min":0.219,"5min":0.16500000000000001,"15min":0.154},"max":{"1min":1.202,"5min":5.7649999999999997,"15min":5.7649999999999997},"last":0.28399999999999997},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.51400000000000001,"15min":0.46899999999999997},"min":{"1min":0.20699999999999999,"5min":0.19,"15min":0.17699999999999999},"max":{"1min":1.585,"5min":9.4770000000000003,"15min":9.4770000000000003},"last":0.26100000000000001}]},{"osd":2,"last update":"Tue Oct 3 17:45:55 2023","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.59999999999999998,"15min":0.53700000000000003},"min":{"1min":0.23200000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":4.1619999999999999,"5min":9.4149999999999991,"15min":9.4149999999999991},"last":0.23000000000000001},{"interface":"front","average":{"1min":1.01,"5min":0.54900000000000004,"15min":0.51400000000000001},"min":{"1min":0.22600000000000001,"5min":0.189,"15min":0.189},"max":{"1min":5.218,"5min":5.218,"15min":5.218},"last":0.41399999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":323584,"data_stored":3511,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":5372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":167936,"data_stored":1861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":578265088,"data_stored":2736168960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":562905088,"data_stored":2843275283,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1140707328,"data_stored":5572378643,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-03T17:46:24.970 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-03T17:34:47.281095+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=34, tm_sec=47, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=40, tm_sec=6, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T17:46:24.970 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T17:46:25.632 INFO:tasks.ceph.osd.0.smithi111.stderr:2023-10-03T17:46:25.635+0000 7ffbac289640 -1 reset not still connected to 0x5563a7ff65b0 2023-10-03T17:46:25.633 INFO:tasks.ceph.osd.0.smithi111.stderr:2023-10-03T17:46:25.635+0000 7ffbac289640 -1 reset not still connected to 0x5563a7ff68f0 2023-10-03T17:46:25.633 INFO:tasks.ceph.osd.0.smithi111.stderr:2023-10-03T17:46:25.635+0000 7ffbac289640 -1 reset not still connected to 0x5563a838e1a0 2023-10-03T17:46:25.633 INFO:tasks.ceph.osd.0.smithi111.stderr:2023-10-03T17:46:25.635+0000 7ffbac289640 -1 reset not still connected to 0x5563a851aea0 2023-10-03T17:46:29.997 INFO:tasks.ceph.mgr.x.smithi111.stderr:2023-10-03T17:46:29.999+0000 7fe48bb97640 -1 calc_pg_upmaps abort due to max <= 0 2023-10-03T17:46:44.971 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T17:46:45.649 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-03T17:46:45.650 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-10-03T17:46:45.688 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":428,"stamp":"2023-10-03T17:46:44.117976+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":21611194159,"num_objects":31497,"num_object_clones":26231,"num_object_copies":62994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31497,"num_whiteouts":0,"num_read":309107,"num_read_kb":1074224,"num_write":285965,"num_write_kb":1142191,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":48651,"ondisk_log_size":48651,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":5470704,"kb_used_data":2279704,"kb_used_omap":4,"kb_used_meta":3190971,"kb_avail":277644816,"statfs":{"total":289910292480,"available":284308291584,"internally_reserved":0,"allocated":2334416896,"data_stored":11307912914,"data_compressed":7460,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4782,"internal_metadata":3267554642},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,4,8,6,15,20,8,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":18063360,"num_objects":-226,"num_object_clones":-226,"num_object_copies":-452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-226,"num_whiteouts":0,"num_read":3646,"num_read_kb":14172,"num_write":3230,"num_write_kb":12920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002027"},"pg_stats":[{"pgid":"2.7","version":"66'1519","reported_seq":2972,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-03T17:46:38.586845+0000","last_change":"2023-10-03T17:40:17.407011+0000","last_active":"2023-10-03T17:46:38.586845+0000","last_peered":"2023-10-03T17:46:38.586845+0000","last_clean":"2023-10-03T17:46:38.586845+0000","last_became_active":"2023-10-03T17:34:42.276624+0000","last_became_peered":"2023-10-03T17:34:42.276624+0000","last_unstale":"2023-10-03T17:46:38.586845+0000","last_undegraded":"2023-10-03T17:46:38.586845+0000","last_fullsized":"2023-10-03T17:46:38.586845+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'673","last_scrub_stamp":"2023-10-03T17:40:17.406813+0000","last_deep_scrub":"46'673","last_deep_scrub_stamp":"2023-10-03T17:40:17.406813+0000","last_clean_scrub_stamp":"2023-10-03T17:40:17.406813+0000","objects_scrubbed":7,"log_size":1519,"log_dups_size":0,"ondisk_log_size":1519,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T21:38:35.056889+0000","scrub_duration":0.013778076,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":3886,"num_read_kb":3202,"num_write":129,"num_write_kb":110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":903,"num_omap_keys":30,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"66'2541","reported_seq":5882,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-03T17:46:38.591806+0000","last_change":"2023-10-03T17:40:13.376519+0000","last_active":"2023-10-03T17:46:38.591806+0000","last_peered":"2023-10-03T17:46:38.591806+0000","last_clean":"2023-10-03T17:46:38.591806+0000","last_became_active":"2023-10-03T17:34:42.271201+0000","last_became_peered":"2023-10-03T17:34:42.271201+0000","last_unstale":"2023-10-03T17:46:38.591806+0000","last_undegraded":"2023-10-03T17:46:38.591806+0000","last_fullsized":"2023-10-03T17:46:38.591806+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'1129","last_scrub_stamp":"2023-10-03T17:40:13.376332+0000","last_deep_scrub":"46'1129","last_deep_scrub_stamp":"2023-10-03T17:40:13.376332+0000","last_clean_scrub_stamp":"2023-10-03T17:40:13.376332+0000","objects_scrubbed":11,"log_size":2541,"log_dups_size":0,"ondisk_log_size":2541,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T18:59:17.088585+0000","scrub_duration":0.015998675,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":674,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":7871,"num_read_kb":6742,"num_write":247,"num_write_kb":217,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2912,"num_omap_keys":93,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"64'33","reported_seq":132,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-10-03T17:46:03.321857+0000","last_change":"2023-10-03T17:40:25.496010+0000","last_active":"2023-10-03T17:46:03.321857+0000","last_peered":"2023-10-03T17:46:03.321857+0000","last_clean":"2023-10-03T17:46:03.321857+0000","last_became_active":"2023-10-03T17:34:42.276941+0000","last_became_peered":"2023-10-03T17:34:42.276941+0000","last_unstale":"2023-10-03T17:46:03.321857+0000","last_undegraded":"2023-10-03T17:46:03.321857+0000","last_fullsized":"2023-10-03T17:46:03.321857+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'14","last_scrub_stamp":"2023-10-03T17:40:25.495827+0000","last_deep_scrub":"46'14","last_deep_scrub_stamp":"2023-10-03T17:40:25.495827+0000","last_clean_scrub_stamp":"2023-10-03T17:40:25.495827+0000","objects_scrubbed":7,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:05:44.939663+0000","scrub_duration":0.0133078,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1007,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":41,"num_read_kb":14,"num_write":26,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"66'2030","reported_seq":4457,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-03T17:46:38.588272+0000","last_change":"2023-10-03T17:40:26.513902+0000","last_active":"2023-10-03T17:46:38.588272+0000","last_peered":"2023-10-03T17:46:38.588272+0000","last_clean":"2023-10-03T17:46:38.588272+0000","last_became_active":"2023-10-03T17:34:42.276459+0000","last_became_peered":"2023-10-03T17:34:42.276459+0000","last_unstale":"2023-10-03T17:46:38.588272+0000","last_undegraded":"2023-10-03T17:46:38.588272+0000","last_fullsized":"2023-10-03T17:46:38.588272+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'964","last_scrub_stamp":"2023-10-03T17:40:26.513729+0000","last_deep_scrub":"46'964","last_deep_scrub_stamp":"2023-10-03T17:40:26.513729+0000","last_clean_scrub_stamp":"2023-10-03T17:40:26.513729+0000","objects_scrubbed":10,"log_size":2030,"log_dups_size":0,"ondisk_log_size":2030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:41:33.182751+0000","scrub_duration":0.018190280999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":384,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":5729,"num_read_kb":4786,"num_write":173,"num_write_kb":154,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1731,"num_omap_keys":80,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"66'84430","reported_seq":149242,"reported_epoch":66,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T17:46:40.696396+0000","last_change":"2023-10-03T17:46:38.860949+0000","last_active":"2023-10-03T17:46:40.696396+0000","last_peered":"2023-10-03T17:46:40.696396+0000","last_clean":"2023-10-03T17:46:40.696396+0000","last_became_active":"2023-10-03T17:34:48.310701+0000","last_became_peered":"2023-10-03T17:34:48.310701+0000","last_unstale":"2023-10-03T17:46:40.696396+0000","last_undegraded":"2023-10-03T17:46:40.696396+0000","last_fullsized":"2023-10-03T17:46:40.696396+0000","mapping_epoch":16,"log_start":"61'74395","ondisk_log_start":"61'74395","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'80695","last_scrub_stamp":"2023-10-03T17:46:13.982571+0000","last_deep_scrub":"66'80695","last_deep_scrub_stamp":"2023-10-03T17:46:13.982571+0000","last_clean_scrub_stamp":"2023-10-03T17:46:13.982571+0000","objects_scrubbed":3410,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":33,"last_scrub_duration":363,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:49:51.684926+0000","scrub_duration":362.26574615599998,"objects_trimmed":507,"snaptrim_duration":33.528732362,"stat_sum":{"num_bytes":5570363392,"num_objects":8519,"num_object_clones":7199,"num_object_copies":17038,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8519,"num_whiteouts":0,"num_read":72329,"num_read_kb":270288,"num_write":73674,"num_write_kb":294696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"54","length":"7"}]},{"pgid":"2.2","version":"66'2051","reported_seq":3982,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-03T17:46:38.006336+0000","last_change":"2023-10-03T17:40:08.675119+0000","last_active":"2023-10-03T17:46:38.006336+0000","last_peered":"2023-10-03T17:46:38.006336+0000","last_clean":"2023-10-03T17:46:38.006336+0000","last_became_active":"2023-10-03T17:34:42.270625+0000","last_became_peered":"2023-10-03T17:34:42.270625+0000","last_unstale":"2023-10-03T17:46:38.006336+0000","last_undegraded":"2023-10-03T17:46:38.006336+0000","last_fullsized":"2023-10-03T17:46:38.006336+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'841","last_scrub_stamp":"2023-10-03T17:40:08.674910+0000","last_deep_scrub":"46'841","last_deep_scrub_stamp":"2023-10-03T17:40:08.674910+0000","last_clean_scrub_stamp":"2023-10-03T17:40:08.674910+0000","objects_scrubbed":11,"log_size":2051,"log_dups_size":0,"ondisk_log_size":2051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T21:05:42.790614+0000","scrub_duration":0.039189555000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":858,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":5201,"num_read_kb":4296,"num_write":185,"num_write_kb":164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2564,"num_omap_keys":60,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"66'78453","reported_seq":139880,"reported_epoch":66,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T17:46:41.029467+0000","last_change":"2023-10-03T17:43:28.494717+0000","last_active":"2023-10-03T17:46:41.029467+0000","last_peered":"2023-10-03T17:46:41.029467+0000","last_clean":"2023-10-03T17:46:41.029467+0000","last_became_active":"2023-10-03T17:34:48.310555+0000","last_became_peered":"2023-10-03T17:34:48.310555+0000","last_unstale":"2023-10-03T17:46:41.029467+0000","last_undegraded":"2023-10-03T17:46:41.029467+0000","last_fullsized":"2023-10-03T17:46:41.029467+0000","mapping_epoch":16,"log_start":"61'68367","ondisk_log_start":"61'68367","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_clean_scrub_stamp":"2023-10-03T17:34:47.281095+0000","objects_scrubbed":3013,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":40,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 192s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086289699999999995,"stat_sum":{"num_bytes":5243441152,"num_objects":8485,"num_object_clones":7243,"num_object_copies":16970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8485,"num_whiteouts":0,"num_read":68328,"num_read_kb":254784,"num_write":68757,"num_write_kb":275028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"66'133","reported_seq":1573,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-03T17:46:38.017957+0000","last_change":"2023-10-03T17:40:17.641493+0000","last_active":"2023-10-03T17:46:38.017957+0000","last_peered":"2023-10-03T17:46:38.017957+0000","last_clean":"2023-10-03T17:46:38.017957+0000","last_became_active":"2023-10-03T17:34:42.278196+0000","last_became_peered":"2023-10-03T17:34:42.278196+0000","last_unstale":"2023-10-03T17:46:38.017957+0000","last_undegraded":"2023-10-03T17:46:38.017957+0000","last_fullsized":"2023-10-03T17:46:38.017957+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'30","last_scrub_stamp":"2023-10-03T17:40:17.640969+0000","last_deep_scrub":"46'30","last_deep_scrub_stamp":"2023-10-03T17:40:17.640969+0000","last_clean_scrub_stamp":"2023-10-03T17:40:17.640969+0000","objects_scrubbed":6,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:04:02.954896+0000","scrub_duration":0.071054565,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":471,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":3867,"num_read_kb":3192,"num_write":116,"num_write_kb":102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":903,"num_omap_keys":30,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"66'83248","reported_seq":145375,"reported_epoch":66,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T17:46:40.686039+0000","last_change":"2023-10-03T17:46:38.335644+0000","last_active":"2023-10-03T17:46:40.686039+0000","last_peered":"2023-10-03T17:46:40.686039+0000","last_clean":"2023-10-03T17:46:40.686039+0000","last_became_active":"2023-10-03T17:34:48.310912+0000","last_became_peered":"2023-10-03T17:34:48.310912+0000","last_unstale":"2023-10-03T17:46:40.686039+0000","last_undegraded":"2023-10-03T17:46:40.686039+0000","last_fullsized":"2023-10-03T17:46:40.686039+0000","mapping_epoch":16,"log_start":"61'73230","ondisk_log_start":"61'73230","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'58196","last_scrub_stamp":"2023-10-03T17:43:26.193075+0000","last_deep_scrub":"53'58196","last_deep_scrub_stamp":"2023-10-03T17:43:26.193075+0000","last_clean_scrub_stamp":"2023-10-03T17:43:26.193075+0000","objects_scrubbed":2326,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":8,"last_scrub_duration":196,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T21:49:35.095231+0000","scrub_duration":195.44739730800001,"objects_trimmed":715,"snaptrim_duration":32.998620240000001,"stat_sum":{"num_bytes":5365850112,"num_objects":6648,"num_object_clones":5377,"num_object_copies":13296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6648,"num_whiteouts":0,"num_read":70116,"num_read_kb":262940,"num_write":70702,"num_write_kb":282808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"54","length":"1e"},{"start":"7c","length":"2"}]},{"pgid":"2.0","version":"64'68","reported_seq":618,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-03T17:46:32.889915+0000","last_change":"2023-10-03T17:40:10.708344+0000","last_active":"2023-10-03T17:46:32.889915+0000","last_peered":"2023-10-03T17:46:32.889915+0000","last_clean":"2023-10-03T17:46:32.889915+0000","last_became_active":"2023-10-03T17:34:42.272306+0000","last_became_peered":"2023-10-03T17:34:42.272306+0000","last_unstale":"2023-10-03T17:46:32.889915+0000","last_undegraded":"2023-10-03T17:46:32.889915+0000","last_fullsized":"2023-10-03T17:46:32.889915+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'20","last_scrub_stamp":"2023-10-03T17:40:10.708060+0000","last_deep_scrub":"46'20","last_deep_scrub_stamp":"2023-10-03T17:40:10.708060+0000","last_clean_scrub_stamp":"2023-10-03T17:40:10.708060+0000","objects_scrubbed":7,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:03:09.097252+0000","scrub_duration":0.042279491000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":805,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":1326,"num_read_kb":1077,"num_write":58,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":301,"num_omap_keys":10,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"66'83822","reported_seq":146456,"reported_epoch":66,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T17:46:41.043314+0000","last_change":"2023-10-03T17:46:40.881810+0000","last_active":"2023-10-03T17:46:41.043314+0000","last_peered":"2023-10-03T17:46:41.043314+0000","last_clean":"2023-10-03T17:46:41.043314+0000","last_became_active":"2023-10-03T17:34:48.310612+0000","last_became_peered":"2023-10-03T17:34:48.310612+0000","last_unstale":"2023-10-03T17:46:41.043314+0000","last_undegraded":"2023-10-03T17:46:41.043314+0000","last_fullsized":"2023-10-03T17:46:41.043314+0000","mapping_epoch":16,"log_start":"61'73817","ondisk_log_start":"61'73817","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'75807","last_scrub_stamp":"2023-10-03T17:45:43.833310+0000","last_deep_scrub":"61'75807","last_deep_scrub_stamp":"2023-10-03T17:45:43.833310+0000","last_clean_scrub_stamp":"2023-10-03T17:45:43.833310+0000","objects_scrubbed":3607,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":22,"last_scrub_duration":315,"scrub_schedule":"queued for deep scrub","scrub_duration":314.26374841099999,"objects_trimmed":656,"snaptrim_duration":35.545564351000003,"stat_sum":{"num_bytes":5430943763,"num_objects":7700,"num_object_clones":6412,"num_object_copies":15400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7700,"num_whiteouts":0,"num_read":70273,"num_read_kb":262796,"num_write":71732,"num_write_kb":286922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"54","length":"12"}]},{"pgid":"2.3","version":"64'33","reported_seq":131,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-10-03T17:46:03.321895+0000","last_change":"2023-10-03T17:40:24.472734+0000","last_active":"2023-10-03T17:46:03.321895+0000","last_peered":"2023-10-03T17:46:03.321895+0000","last_clean":"2023-10-03T17:46:03.321895+0000","last_became_active":"2023-10-03T17:34:42.270194+0000","last_became_peered":"2023-10-03T17:34:42.270194+0000","last_unstale":"2023-10-03T17:46:03.321895+0000","last_undegraded":"2023-10-03T17:46:03.321895+0000","last_fullsized":"2023-10-03T17:46:03.321895+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'8","last_scrub_stamp":"2023-10-03T17:40:24.472573+0000","last_deep_scrub":"46'8","last_deep_scrub_stamp":"2023-10-03T17:40:24.472573+0000","last_clean_scrub_stamp":"2023-10-03T17:40:24.472573+0000","objects_scrubbed":4,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T18:28:04.735033+0000","scrub_duration":0.024990637999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":585,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":34,"num_read_kb":16,"num_write":29,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"46'99","reported_seq":208,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-10-03T17:46:03.322956+0000","last_change":"2023-10-03T17:40:10.452341+0000","last_active":"2023-10-03T17:46:03.322956+0000","last_peered":"2023-10-03T17:46:03.322956+0000","last_clean":"2023-10-03T17:46:03.322956+0000","last_became_active":"2023-10-03T17:34:39.502393+0000","last_became_peered":"2023-10-03T17:34:39.502393+0000","last_unstale":"2023-10-03T17:46:03.322956+0000","last_undegraded":"2023-10-03T17:46:03.322956+0000","last_fullsized":"2023-10-03T17:46:03.322956+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'99","last_scrub_stamp":"2023-10-03T17:40:10.449099+0000","last_deep_scrub":"46'99","last_deep_scrub_stamp":"2023-10-03T17:40:10.449099+0000","last_clean_scrub_stamp":"2023-10-03T17:40:10.449099+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T20:18:04.958666+0000","scrub_duration":0.043225012,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":21610598419,"num_objects":31352,"num_object_clones":26231,"num_object_copies":62704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31352,"num_whiteouts":0,"num_read":281046,"num_read_kb":1050808,"num_write":284865,"num_write_kb":1139454,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2333270016,"data_stored":11299569702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40144,"ondisk_log_size":40144,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":5372,"num_objects":143,"num_object_clones":0,"num_object_copies":286,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":143,"num_whiteouts":0,"num_read":27955,"num_read_kb":23325,"num_write":963,"num_write_kb":843,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":983040,"data_stored":10744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8408,"ondisk_log_size":8408,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":7460,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738519,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2680772,"kb_used_data":1138812,"kb_used_omap":1,"kb_used_meta":1541950,"kb_avail":91691068,"statfs":{"total":96636764160,"available":93891653632,"internally_reserved":0,"allocated":1166143488,"data_stored":5655280852,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1578957254},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,4,2,4,10],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 17:45:55 2023","interfaces":[{"interface":"back","average":{"1min":0.30099999999999999,"5min":0.40899999999999997,"15min":0.442},"min":{"1min":0.17999999999999999,"5min":0.158,"15min":0.14299999999999999},"max":{"1min":0.51100000000000001,"5min":4.1539999999999999,"15min":4.4640000000000004},"last":0.22500000000000001},{"interface":"front","average":{"1min":0.312,"5min":0.42699999999999999,"15min":0.45500000000000002},"min":{"1min":0.19700000000000001,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":0.54000000000000004,"5min":4.4180000000000001,"15min":4.4180000000000001},"last":0.29099999999999998}]},{"osd":1,"last update":"Tue Oct 3 17:45:55 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.61299999999999999,"15min":0.52700000000000002},"min":{"1min":0.184,"5min":0.17599999999999999,"15min":0.16600000000000001},"max":{"1min":4.0519999999999996,"5min":8.1059999999999999,"15min":8.1059999999999999},"last":0.313},{"interface":"front","average":{"1min":0.76100000000000001,"5min":0.501,"15min":0.49399999999999999},"min":{"1min":0.248,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":4.1120000000000001,"5min":4.1120000000000001,"15min":4.1120000000000001},"last":0.255}]}]},{"osd":1,"up_from":8,"seq":34359738517,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1399024,"kb_used_data":562728,"kb_used_omap":1,"kb_used_meta":836286,"kb_avail":92972816,"statfs":{"total":96636764160,"available":95204163584,"internally_reserved":0,"allocated":576233472,"data_stored":2925051051,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":856357318},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,2,0,5,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 17:46:09 2023","interfaces":[{"interface":"back","average":{"1min":0.33200000000000002,"5min":0.33000000000000002,"15min":0.40100000000000002},"min":{"1min":0.19800000000000001,"5min":0.17199999999999999,"15min":0.161},"max":{"1min":0.88300000000000001,"5min":2.3809999999999998,"15min":4.1879999999999997},"last":0.23000000000000001},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.45100000000000001,"15min":0.443},"min":{"1min":0.17699999999999999,"5min":0.17599999999999999,"15min":0.17399999999999999},"max":{"1min":2.4809999999999999,"5min":4.5759999999999996,"15min":4.5759999999999996},"last":0.19800000000000001}]},{"osd":2,"last update":"Tue Oct 3 17:46:09 2023","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.38,"15min":0.432},"min":{"1min":0.21199999999999999,"5min":0.187,"15min":0.159},"max":{"1min":1.026,"5min":2.6110000000000002,"15min":3.0390000000000001},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.42099999999999999,"5min":0.34300000000000003,"15min":0.433},"min":{"1min":0.26800000000000002,"5min":0.16400000000000001,"15min":0.16400000000000001},"max":{"1min":0.79100000000000004,"5min":1.337,"15min":3.0699999999999998},"last":0.32500000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738517,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1390908,"kb_used_data":578164,"kb_used_omap":1,"kb_used_meta":812734,"kb_avail":92980932,"statfs":{"total":96636764160,"available":95212474368,"internally_reserved":0,"allocated":592039936,"data_stored":2727581011,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":832240070},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,2,4,6,6,8,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 17:45:55 2023","interfaces":[{"interface":"back","average":{"1min":0.371,"5min":0.42999999999999999,"15min":0.42199999999999999},"min":{"1min":0.219,"5min":0.16500000000000001,"15min":0.154},"max":{"1min":1.202,"5min":5.7649999999999997,"15min":5.7649999999999997},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.51400000000000001,"15min":0.46899999999999997},"min":{"1min":0.20699999999999999,"5min":0.19,"15min":0.17699999999999999},"max":{"1min":1.585,"5min":9.4770000000000003,"15min":9.4770000000000003},"last":0.30499999999999999}]},{"osd":2,"last update":"Tue Oct 3 17:45:55 2023","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.59999999999999998,"15min":0.53700000000000003},"min":{"1min":0.23200000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":4.1619999999999999,"5min":9.4149999999999991,"15min":9.4149999999999991},"last":0.27100000000000002},{"interface":"front","average":{"1min":1.01,"5min":0.54900000000000004,"15min":0.51400000000000001},"min":{"1min":0.22600000000000001,"5min":0.189,"15min":0.189},"max":{"1min":5.218,"5min":5.218,"15min":5.218},"last":0.33400000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":323584,"data_stored":3511,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":5372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":167936,"data_stored":1861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":591364096,"data_stored":2724741120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":575451136,"data_stored":2925043731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1166454784,"data_stored":5649784851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-03T17:46:45.691 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-03T17:34:47.281095+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=34, tm_sec=47, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=40, tm_sec=6, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T17:46:45.691 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T17:47:05.692 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T17:47:06.180 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-03T17:47:06.181 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-10-03T17:47:06.206 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":443,"stamp":"2023-10-03T17:47:06.121431+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":21650134831,"num_objects":30877,"num_object_clones":25611,"num_object_copies":61754,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30877,"num_whiteouts":0,"num_read":319118,"num_read_kb":1107070,"num_write":293461,"num_write_kb":1171723,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":49027,"ondisk_log_size":49027,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":5681536,"kb_used_data":2341984,"kb_used_omap":4,"kb_used_meta":3339515,"kb_avail":277433984,"statfs":{"total":289910292480,"available":284092399616,"internally_reserved":0,"allocated":2398191616,"data_stored":11249966565,"data_compressed":7460,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4782,"internal_metadata":3419663698},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,2,4,15,19,23,14,0,1,2,2],"upper_bound":8192},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":12427264,"num_objects":171,"num_object_clones":171,"num_object_copies":342,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":171,"num_whiteouts":0,"num_read":4503,"num_read_kb":11402,"num_write":2596,"num_write_kb":9932,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.802034"},"pg_stats":[{"pgid":"2.7","version":"67'1565","reported_seq":3087,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-03T17:47:01.024505+0000","last_change":"2023-10-03T17:40:17.407011+0000","last_active":"2023-10-03T17:47:01.024505+0000","last_peered":"2023-10-03T17:47:01.024505+0000","last_clean":"2023-10-03T17:47:01.024505+0000","last_became_active":"2023-10-03T17:34:42.276624+0000","last_became_peered":"2023-10-03T17:34:42.276624+0000","last_unstale":"2023-10-03T17:47:01.024505+0000","last_undegraded":"2023-10-03T17:47:01.024505+0000","last_fullsized":"2023-10-03T17:47:01.024505+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'673","last_scrub_stamp":"2023-10-03T17:40:17.406813+0000","last_deep_scrub":"46'673","last_deep_scrub_stamp":"2023-10-03T17:40:17.406813+0000","last_clean_scrub_stamp":"2023-10-03T17:40:17.406813+0000","objects_scrubbed":7,"log_size":1565,"log_dups_size":0,"ondisk_log_size":1565,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T21:38:35.056889+0000","scrub_duration":0.013778076,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":688,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":4154,"num_read_kb":3420,"num_write":143,"num_write_kb":124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":903,"num_omap_keys":30,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"67'2613","reported_seq":6072,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-03T17:47:01.026155+0000","last_change":"2023-10-03T17:40:13.376519+0000","last_active":"2023-10-03T17:47:01.026155+0000","last_peered":"2023-10-03T17:47:01.026155+0000","last_clean":"2023-10-03T17:47:01.026155+0000","last_became_active":"2023-10-03T17:34:42.271201+0000","last_became_peered":"2023-10-03T17:34:42.271201+0000","last_unstale":"2023-10-03T17:47:01.026155+0000","last_undegraded":"2023-10-03T17:47:01.026155+0000","last_fullsized":"2023-10-03T17:47:01.026155+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'1129","last_scrub_stamp":"2023-10-03T17:40:13.376332+0000","last_deep_scrub":"46'1129","last_deep_scrub_stamp":"2023-10-03T17:40:13.376332+0000","last_clean_scrub_stamp":"2023-10-03T17:40:13.376332+0000","objects_scrubbed":11,"log_size":2613,"log_dups_size":0,"ondisk_log_size":2613,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T18:59:17.088585+0000","scrub_duration":0.015998675,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":774,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":8180,"num_read_kb":7001,"num_write":263,"num_write_kb":232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2912,"num_omap_keys":93,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"67'34","reported_seq":135,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-03T17:47:00.516449+0000","last_change":"2023-10-03T17:40:25.496010+0000","last_active":"2023-10-03T17:47:00.516449+0000","last_peered":"2023-10-03T17:47:00.516449+0000","last_clean":"2023-10-03T17:47:00.516449+0000","last_became_active":"2023-10-03T17:34:42.276941+0000","last_became_peered":"2023-10-03T17:34:42.276941+0000","last_unstale":"2023-10-03T17:47:00.516449+0000","last_undegraded":"2023-10-03T17:47:00.516449+0000","last_fullsized":"2023-10-03T17:47:00.516449+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'14","last_scrub_stamp":"2023-10-03T17:40:25.495827+0000","last_deep_scrub":"46'14","last_deep_scrub_stamp":"2023-10-03T17:40:25.495827+0000","last_clean_scrub_stamp":"2023-10-03T17:40:25.495827+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:05:44.939663+0000","scrub_duration":0.0133078,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":957,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":42,"num_read_kb":15,"num_write":27,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"67'2088","reported_seq":4610,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-03T17:47:01.025799+0000","last_change":"2023-10-03T17:40:26.513902+0000","last_active":"2023-10-03T17:47:01.025799+0000","last_peered":"2023-10-03T17:47:01.025799+0000","last_clean":"2023-10-03T17:47:01.025799+0000","last_became_active":"2023-10-03T17:34:42.276459+0000","last_became_peered":"2023-10-03T17:34:42.276459+0000","last_unstale":"2023-10-03T17:47:01.025799+0000","last_undegraded":"2023-10-03T17:47:01.025799+0000","last_fullsized":"2023-10-03T17:47:01.025799+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'964","last_scrub_stamp":"2023-10-03T17:40:26.513729+0000","last_deep_scrub":"46'964","last_deep_scrub_stamp":"2023-10-03T17:40:26.513729+0000","last_clean_scrub_stamp":"2023-10-03T17:40:26.513729+0000","objects_scrubbed":10,"log_size":2088,"log_dups_size":0,"ondisk_log_size":2088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:41:33.182751+0000","scrub_duration":0.018190280999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":284,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":6019,"num_read_kb":5026,"num_write":187,"num_write_kb":166,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1731,"num_omap_keys":80,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"70'88015","reported_seq":154553,"reported_epoch":70,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T17:47:05.879517+0000","last_change":"2023-10-03T17:47:05.658372+0000","last_active":"2023-10-03T17:47:05.879517+0000","last_peered":"2023-10-03T17:47:05.879517+0000","last_clean":"2023-10-03T17:47:05.879517+0000","last_became_active":"2023-10-03T17:34:48.310701+0000","last_became_peered":"2023-10-03T17:34:48.310701+0000","last_unstale":"2023-10-03T17:47:05.879517+0000","last_undegraded":"2023-10-03T17:47:05.879517+0000","last_fullsized":"2023-10-03T17:47:05.879517+0000","mapping_epoch":16,"log_start":"61'77915","ondisk_log_start":"61'77915","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'80695","last_scrub_stamp":"2023-10-03T17:46:13.982571+0000","last_deep_scrub":"66'80695","last_deep_scrub_stamp":"2023-10-03T17:46:13.982571+0000","last_clean_scrub_stamp":"2023-10-03T17:46:13.982571+0000","objects_scrubbed":3410,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":43,"last_scrub_duration":363,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T21:31:03.424497+0000","scrub_duration":362.26574615599998,"objects_trimmed":51,"snaptrim_duration":2.2956227469999999,"stat_sum":{"num_bytes":5581660160,"num_objects":8279,"num_object_clones":6959,"num_object_copies":16558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8279,"num_whiteouts":0,"num_read":74657,"num_read_kb":279372,"num_write":75707,"num_write_kb":302828,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"54","length":"11"}]},{"pgid":"2.2","version":"70'2135","reported_seq":4227,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-03T17:47:03.606884+0000","last_change":"2023-10-03T17:40:08.675119+0000","last_active":"2023-10-03T17:47:03.606884+0000","last_peered":"2023-10-03T17:47:03.606884+0000","last_clean":"2023-10-03T17:47:03.606884+0000","last_became_active":"2023-10-03T17:34:42.270625+0000","last_became_peered":"2023-10-03T17:34:42.270625+0000","last_unstale":"2023-10-03T17:47:03.606884+0000","last_undegraded":"2023-10-03T17:47:03.606884+0000","last_fullsized":"2023-10-03T17:47:03.606884+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'841","last_scrub_stamp":"2023-10-03T17:40:08.674910+0000","last_deep_scrub":"46'841","last_deep_scrub_stamp":"2023-10-03T17:40:08.674910+0000","last_clean_scrub_stamp":"2023-10-03T17:40:08.674910+0000","objects_scrubbed":11,"log_size":2135,"log_dups_size":0,"ondisk_log_size":2135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T21:05:42.790614+0000","scrub_duration":0.039189555000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":959,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":5776,"num_read_kb":4775,"num_write":229,"num_write_kb":197,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2564,"num_omap_keys":60,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"67'80483","reported_seq":143538,"reported_epoch":67,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T17:47:01.252606+0000","last_change":"2023-10-03T17:43:28.494717+0000","last_active":"2023-10-03T17:47:01.252606+0000","last_peered":"2023-10-03T17:47:01.252606+0000","last_clean":"2023-10-03T17:47:01.252606+0000","last_became_active":"2023-10-03T17:34:48.310555+0000","last_became_peered":"2023-10-03T17:34:48.310555+0000","last_unstale":"2023-10-03T17:47:01.252606+0000","last_undegraded":"2023-10-03T17:47:01.252606+0000","last_fullsized":"2023-10-03T17:47:01.252606+0000","mapping_epoch":16,"log_start":"61'70467","ondisk_log_start":"61'70467","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_clean_scrub_stamp":"2023-10-03T17:34:47.281095+0000","objects_scrubbed":3344,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":40,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 212s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086289699999999995,"stat_sum":{"num_bytes":5251399680,"num_objects":8627,"num_object_clones":7385,"num_object_copies":17254,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8627,"num_whiteouts":0,"num_read":70086,"num_read_kb":261588,"num_write":70356,"num_write_kb":281424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"70'153","reported_seq":1702,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-03T17:47:03.971972+0000","last_change":"2023-10-03T17:40:17.641493+0000","last_active":"2023-10-03T17:47:03.971972+0000","last_peered":"2023-10-03T17:47:03.971972+0000","last_clean":"2023-10-03T17:47:03.971972+0000","last_became_active":"2023-10-03T17:34:42.278196+0000","last_became_peered":"2023-10-03T17:34:42.278196+0000","last_unstale":"2023-10-03T17:47:03.971972+0000","last_undegraded":"2023-10-03T17:47:03.971972+0000","last_fullsized":"2023-10-03T17:47:03.971972+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'30","last_scrub_stamp":"2023-10-03T17:40:17.640969+0000","last_deep_scrub":"46'30","last_deep_scrub_stamp":"2023-10-03T17:40:17.640969+0000","last_clean_scrub_stamp":"2023-10-03T17:40:17.640969+0000","objects_scrubbed":6,"log_size":153,"log_dups_size":0,"ondisk_log_size":153,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:04:02.954896+0000","scrub_duration":0.071054565,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":470,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":4290,"num_read_kb":3543,"num_write":147,"num_write_kb":125,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":903,"num_omap_keys":30,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"70'86713","reported_seq":150361,"reported_epoch":70,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T17:47:05.883279+0000","last_change":"2023-10-03T17:47:05.601255+0000","last_active":"2023-10-03T17:47:05.883279+0000","last_peered":"2023-10-03T17:47:05.883279+0000","last_clean":"2023-10-03T17:47:05.883279+0000","last_became_active":"2023-10-03T17:34:48.310912+0000","last_became_peered":"2023-10-03T17:34:48.310912+0000","last_unstale":"2023-10-03T17:47:05.883279+0000","last_undegraded":"2023-10-03T17:47:05.883279+0000","last_fullsized":"2023-10-03T17:47:05.883279+0000","mapping_epoch":16,"log_start":"61'76648","ondisk_log_start":"61'76648","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'58196","last_scrub_stamp":"2023-10-03T17:43:26.193075+0000","last_deep_scrub":"53'58196","last_deep_scrub_stamp":"2023-10-03T17:43:26.193075+0000","last_clean_scrub_stamp":"2023-10-03T17:43:26.193075+0000","objects_scrubbed":2326,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":19,"last_scrub_duration":196,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:57:33.401519+0000","scrub_duration":195.44739730800001,"objects_trimmed":29,"snaptrim_duration":2.2210716740000001,"stat_sum":{"num_bytes":5376512000,"num_objects":6522,"num_object_clones":5251,"num_object_copies":13044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6522,"num_whiteouts":0,"num_read":72230,"num_read_kb":271188,"num_write":72775,"num_write_kb":291100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"54","length":"1e"},{"start":"7c","length":"8"},{"start":"8a","length":"2"},{"start":"90","length":"1"}]},{"pgid":"2.0","version":"69'81","reported_seq":678,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-03T17:47:03.450558+0000","last_change":"2023-10-03T17:40:10.708344+0000","last_active":"2023-10-03T17:47:03.450558+0000","last_peered":"2023-10-03T17:47:03.450558+0000","last_clean":"2023-10-03T17:47:03.450558+0000","last_became_active":"2023-10-03T17:34:42.272306+0000","last_became_peered":"2023-10-03T17:34:42.272306+0000","last_unstale":"2023-10-03T17:47:03.450558+0000","last_undegraded":"2023-10-03T17:47:03.450558+0000","last_fullsized":"2023-10-03T17:47:03.450558+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'20","last_scrub_stamp":"2023-10-03T17:40:10.708060+0000","last_deep_scrub":"46'20","last_deep_scrub_stamp":"2023-10-03T17:40:10.708060+0000","last_clean_scrub_stamp":"2023-10-03T17:40:10.708060+0000","objects_scrubbed":7,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:03:09.097252+0000","scrub_duration":0.042279491000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":855,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1472,"num_read_kb":1199,"num_write":74,"num_write_kb":62,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":301,"num_omap_keys":10,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"67'86571","reported_seq":150686,"reported_epoch":67,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T17:47:01.270975+0000","last_change":"2023-10-03T17:47:00.726414+0000","last_active":"2023-10-03T17:47:01.270975+0000","last_peered":"2023-10-03T17:47:01.270975+0000","last_clean":"2023-10-03T17:47:01.270975+0000","last_became_active":"2023-10-03T17:34:48.310612+0000","last_became_peered":"2023-10-03T17:34:48.310612+0000","last_unstale":"2023-10-03T17:47:01.270975+0000","last_undegraded":"2023-10-03T17:47:01.270975+0000","last_fullsized":"2023-10-03T17:47:01.270975+0000","mapping_epoch":16,"log_start":"61'76530","ondisk_log_start":"61'76530","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'75807","last_scrub_stamp":"2023-10-03T17:45:43.833310+0000","last_deep_scrub":"61'75807","last_deep_scrub_stamp":"2023-10-03T17:45:43.833310+0000","last_clean_scrub_stamp":"2023-10-03T17:45:43.833310+0000","objects_scrubbed":3607,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":14,"last_scrub_duration":315,"scrub_schedule":"queued for deep scrub","scrub_duration":314.26374841099999,"objects_trimmed":24,"snaptrim_duration":0.40022885899999999,"stat_sum":{"num_bytes":5439967251,"num_objects":7304,"num_object_clones":6016,"num_object_copies":14608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7304,"num_whiteouts":0,"num_read":72068,"num_read_kb":269832,"num_write":73383,"num_write_kb":293526,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"54","length":"1a"}]},{"pgid":"2.3","version":"67'37","reported_seq":140,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-03T17:47:00.825265+0000","last_change":"2023-10-03T17:40:24.472734+0000","last_active":"2023-10-03T17:47:00.825265+0000","last_peered":"2023-10-03T17:47:00.825265+0000","last_clean":"2023-10-03T17:47:00.825265+0000","last_became_active":"2023-10-03T17:34:42.270194+0000","last_became_peered":"2023-10-03T17:34:42.270194+0000","last_unstale":"2023-10-03T17:47:00.825265+0000","last_undegraded":"2023-10-03T17:47:00.825265+0000","last_fullsized":"2023-10-03T17:47:00.825265+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'8","last_scrub_stamp":"2023-10-03T17:40:24.472573+0000","last_deep_scrub":"46'8","last_deep_scrub_stamp":"2023-10-03T17:40:24.472573+0000","last_clean_scrub_stamp":"2023-10-03T17:40:24.472573+0000","objects_scrubbed":4,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T18:28:04.735033+0000","scrub_duration":0.024990637999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":385,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":38,"num_read_kb":20,"num_write":33,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"46'99","reported_seq":210,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-03T17:46:46.134196+0000","last_change":"2023-10-03T17:40:10.452341+0000","last_active":"2023-10-03T17:46:46.134196+0000","last_peered":"2023-10-03T17:46:46.134196+0000","last_clean":"2023-10-03T17:46:46.134196+0000","last_became_active":"2023-10-03T17:34:39.502393+0000","last_became_peered":"2023-10-03T17:34:39.502393+0000","last_unstale":"2023-10-03T17:46:46.134196+0000","last_undegraded":"2023-10-03T17:46:46.134196+0000","last_fullsized":"2023-10-03T17:46:46.134196+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'99","last_scrub_stamp":"2023-10-03T17:40:10.449099+0000","last_deep_scrub":"46'99","last_deep_scrub_stamp":"2023-10-03T17:40:10.449099+0000","last_clean_scrub_stamp":"2023-10-03T17:40:10.449099+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T20:18:04.958666+0000","scrub_duration":0.043225012,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":21649539091,"num_objects":30732,"num_object_clones":25611,"num_object_copies":61464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30732,"num_whiteouts":0,"num_read":289041,"num_read_kb":1081980,"num_write":292221,"num_write_kb":1168878,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2396393472,"data_stored":11275935782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40222,"ondisk_log_size":40222,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":5372,"num_objects":143,"num_object_clones":0,"num_object_copies":286,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":143,"num_whiteouts":0,"num_read":29971,"num_read_kb":24999,"num_write":1103,"num_write_kb":951,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":983040,"data_stored":10744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8706,"ondisk_log_size":8706,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":7460,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738524,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2822188,"kb_used_data":1171940,"kb_used_omap":1,"kb_used_meta":1650238,"kb_avail":91549652,"statfs":{"total":96636764160,"available":93746843648,"internally_reserved":0,"allocated":1200066560,"data_stored":5679569924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1689844166},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,2,6,9,17,4,0,1,2,2],"upper_bound":8192},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 17:47:00 2023","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.436},"min":{"1min":0.17199999999999999,"5min":0.16500000000000001,"15min":0.14299999999999999},"max":{"1min":3.6539999999999999,"5min":4.1539999999999999,"15min":4.4640000000000004},"last":0.17199999999999999},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.42399999999999999,"15min":0.44500000000000001},"min":{"1min":0.17999999999999999,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":1.758,"5min":4.4180000000000001,"15min":4.4180000000000001},"last":0.217}]},{"osd":1,"last update":"Tue Oct 3 17:47:00 2023","interfaces":[{"interface":"back","average":{"1min":0.32400000000000001,"5min":0.61899999999999999,"15min":0.51200000000000001},"min":{"1min":0.221,"5min":0.184,"15min":0.16600000000000001},"max":{"1min":0.66600000000000004,"5min":8.1059999999999999,"15min":8.1059999999999999},"last":0.26200000000000001},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.52900000000000003,"15min":0.48399999999999999},"min":{"1min":0.223,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":1.627,"5min":4.1120000000000001,"15min":4.1120000000000001},"last":0.34399999999999997}]}]},{"osd":1,"up_from":8,"seq":34359738521,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1429928,"kb_used_data":575580,"kb_used_omap":1,"kb_used_meta":854334,"kb_avail":92941912,"statfs":{"total":96636764160,"available":95172517888,"internally_reserved":0,"allocated":589393920,"data_stored":2886241430,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":874838470},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,7,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 17:46:09 2023","interfaces":[{"interface":"back","average":{"1min":0.33200000000000002,"5min":0.33000000000000002,"15min":0.40100000000000002},"min":{"1min":0.19800000000000001,"5min":0.17199999999999999,"15min":0.161},"max":{"1min":0.88300000000000001,"5min":2.3809999999999998,"15min":4.1879999999999997},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.45100000000000001,"15min":0.443},"min":{"1min":0.17699999999999999,"5min":0.17599999999999999,"15min":0.17399999999999999},"max":{"1min":2.4809999999999999,"5min":4.5759999999999996,"15min":4.5759999999999996},"last":0.45100000000000001}]},{"osd":2,"last update":"Tue Oct 3 17:46:09 2023","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.38,"15min":0.432},"min":{"1min":0.21199999999999999,"5min":0.187,"15min":0.159},"max":{"1min":1.026,"5min":2.6110000000000002,"15min":3.0390000000000001},"last":0.251},{"interface":"front","average":{"1min":0.42099999999999999,"5min":0.34300000000000003,"15min":0.433},"min":{"1min":0.26800000000000002,"5min":0.16400000000000001,"15min":0.16400000000000001},"max":{"1min":0.79100000000000004,"5min":1.337,"15min":3.0699999999999998},"last":0.32800000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738522,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1429420,"kb_used_data":594464,"kb_used_omap":1,"kb_used_meta":834942,"kb_avail":92942420,"statfs":{"total":96636764160,"available":95173038080,"internally_reserved":0,"allocated":608731136,"data_stored":2684155211,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":854981062},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1,2,6,6,10],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 17:46:57 2023","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.38800000000000001,"15min":0.42799999999999999},"min":{"1min":0.182,"5min":0.17100000000000001,"15min":0.154},"max":{"1min":3.4649999999999999,"5min":3.4649999999999999,"15min":5.7649999999999997},"last":0.187},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.54200000000000004,"15min":0.47099999999999997},"min":{"1min":0.16900000000000001,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":3.4279999999999999,"5min":9.4770000000000003,"15min":9.4770000000000003},"last":0.247}]},{"osd":2,"last update":"Tue Oct 3 17:46:57 2023","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.66400000000000003,"15min":0.54100000000000004},"min":{"1min":0.20000000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":3.4540000000000002,"5min":9.4149999999999991,"15min":9.4149999999999991},"last":0.216},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.61499999999999999,"15min":0.52100000000000002},"min":{"1min":0.253,"5min":0.189,"15min":0.189},"max":{"1min":3.4249999999999998,"5min":5.218,"15min":5.218},"last":0.35399999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":3611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":5372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":159744,"data_stored":1761,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":608186368,"data_stored":2697056256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":588750848,"data_stored":2893692947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1199456256,"data_stored":5685186579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-03T17:47:06.208 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-03T17:34:47.281095+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=34, tm_sec=47, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=40, tm_sec=6, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T17:47:06.208 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T17:47:26.210 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T17:47:26.691 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-03T17:47:26.691 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-10-03T17:47:26.733 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":453,"stamp":"2023-10-03T17:47:26.124783+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":21679789871,"num_objects":32862,"num_object_clones":27596,"num_object_copies":65724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32862,"num_whiteouts":0,"num_read":327163,"num_read_kb":1131131,"num_write":299408,"num_write_kb":1195243,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":49063,"ondisk_log_size":49063,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":5867244,"kb_used_data":2383316,"kb_used_omap":4,"kb_used_meta":3483899,"kb_avail":277248276,"statfs":{"total":289910292480,"available":283902234624,"internally_reserved":0,"allocated":2440515584,"data_stored":12176362894,"data_compressed":7460,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4782,"internal_metadata":3567512914},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,10,7,6,21,21],"upper_bound":256},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":18681856,"num_objects":816,"num_object_clones":816,"num_object_copies":1632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":816,"num_whiteouts":0,"num_read":4910,"num_read_kb":14787,"num_write":3754,"num_write_kb":15016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001939"},"pg_stats":[{"pgid":"2.7","version":"70'1581","reported_seq":3195,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-03T17:47:18.966153+0000","last_change":"2023-10-03T17:40:17.407011+0000","last_active":"2023-10-03T17:47:18.966153+0000","last_peered":"2023-10-03T17:47:18.966153+0000","last_clean":"2023-10-03T17:47:18.966153+0000","last_became_active":"2023-10-03T17:34:42.276624+0000","last_became_peered":"2023-10-03T17:34:42.276624+0000","last_unstale":"2023-10-03T17:47:18.966153+0000","last_undegraded":"2023-10-03T17:47:18.966153+0000","last_fullsized":"2023-10-03T17:47:18.966153+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'673","last_scrub_stamp":"2023-10-03T17:40:17.406813+0000","last_deep_scrub":"46'673","last_deep_scrub_stamp":"2023-10-03T17:40:17.406813+0000","last_clean_scrub_stamp":"2023-10-03T17:40:17.406813+0000","objects_scrubbed":7,"log_size":1581,"log_dups_size":0,"ondisk_log_size":1581,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T21:38:35.056889+0000","scrub_duration":0.013778076,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":739,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":4514,"num_read_kb":3722,"num_write":161,"num_write_kb":135,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":903,"num_omap_keys":30,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"70'2641","reported_seq":6373,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-03T17:47:18.967514+0000","last_change":"2023-10-03T17:40:13.376519+0000","last_active":"2023-10-03T17:47:18.967514+0000","last_peered":"2023-10-03T17:47:18.967514+0000","last_clean":"2023-10-03T17:47:18.967514+0000","last_became_active":"2023-10-03T17:34:42.271201+0000","last_became_peered":"2023-10-03T17:34:42.271201+0000","last_unstale":"2023-10-03T17:47:18.967514+0000","last_undegraded":"2023-10-03T17:47:18.967514+0000","last_fullsized":"2023-10-03T17:47:18.967514+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'1129","last_scrub_stamp":"2023-10-03T17:40:13.376332+0000","last_deep_scrub":"46'1129","last_deep_scrub_stamp":"2023-10-03T17:40:13.376332+0000","last_clean_scrub_stamp":"2023-10-03T17:40:13.376332+0000","objects_scrubbed":11,"log_size":2641,"log_dups_size":0,"ondisk_log_size":2641,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T18:59:17.088585+0000","scrub_duration":0.015998675,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":723,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":9082,"num_read_kb":7772,"num_write":297,"num_write_kb":255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2912,"num_omap_keys":93,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"69'36","reported_seq":142,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-03T17:47:03.351077+0000","last_change":"2023-10-03T17:40:25.496010+0000","last_active":"2023-10-03T17:47:03.351077+0000","last_peered":"2023-10-03T17:47:03.351077+0000","last_clean":"2023-10-03T17:47:03.351077+0000","last_became_active":"2023-10-03T17:34:42.276941+0000","last_became_peered":"2023-10-03T17:34:42.276941+0000","last_unstale":"2023-10-03T17:47:03.351077+0000","last_undegraded":"2023-10-03T17:47:03.351077+0000","last_fullsized":"2023-10-03T17:47:03.351077+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'14","last_scrub_stamp":"2023-10-03T17:40:25.495827+0000","last_deep_scrub":"46'14","last_deep_scrub_stamp":"2023-10-03T17:40:25.495827+0000","last_clean_scrub_stamp":"2023-10-03T17:40:25.495827+0000","objects_scrubbed":7,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:05:44.939663+0000","scrub_duration":0.0133078,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":957,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":43,"num_read_kb":16,"num_write":29,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"70'2109","reported_seq":4816,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-03T17:47:18.967380+0000","last_change":"2023-10-03T17:40:26.513902+0000","last_active":"2023-10-03T17:47:18.967380+0000","last_peered":"2023-10-03T17:47:18.967380+0000","last_clean":"2023-10-03T17:47:18.967380+0000","last_became_active":"2023-10-03T17:34:42.276459+0000","last_became_peered":"2023-10-03T17:34:42.276459+0000","last_unstale":"2023-10-03T17:47:18.967380+0000","last_undegraded":"2023-10-03T17:47:18.967380+0000","last_fullsized":"2023-10-03T17:47:18.967380+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'964","last_scrub_stamp":"2023-10-03T17:40:26.513729+0000","last_deep_scrub":"46'964","last_deep_scrub_stamp":"2023-10-03T17:40:26.513729+0000","last_clean_scrub_stamp":"2023-10-03T17:40:26.513729+0000","objects_scrubbed":10,"log_size":2109,"log_dups_size":0,"ondisk_log_size":2109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:41:33.182751+0000","scrub_duration":0.018190280999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":6630,"num_read_kb":5542,"num_write":212,"num_write_kb":182,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1731,"num_omap_keys":80,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"70'90894","reported_seq":158059,"reported_epoch":70,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T17:47:26.039285+0000","last_change":"2023-10-03T17:47:23.653540+0000","last_active":"2023-10-03T17:47:26.039285+0000","last_peered":"2023-10-03T17:47:26.039285+0000","last_clean":"2023-10-03T17:47:26.039285+0000","last_became_active":"2023-10-03T17:34:48.310701+0000","last_became_peered":"2023-10-03T17:34:48.310701+0000","last_unstale":"2023-10-03T17:47:26.039285+0000","last_undegraded":"2023-10-03T17:47:26.039285+0000","last_fullsized":"2023-10-03T17:47:26.039285+0000","mapping_epoch":16,"log_start":"66'80831","ondisk_log_start":"66'80831","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'80695","last_scrub_stamp":"2023-10-03T17:46:13.982571+0000","last_deep_scrub":"66'80695","last_deep_scrub_stamp":"2023-10-03T17:46:13.982571+0000","last_clean_scrub_stamp":"2023-10-03T17:46:13.982571+0000","objects_scrubbed":3410,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":39,"last_scrub_duration":363,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T21:31:03.424497+0000","scrub_duration":362.26574615599998,"objects_trimmed":349,"snaptrim_duration":20.290467274000001,"stat_sum":{"num_bytes":5589716992,"num_objects":8704,"num_object_clones":7384,"num_object_copies":17408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8704,"num_whiteouts":0,"num_read":76135,"num_read_kb":285128,"num_write":77267,"num_write_kb":309068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"54","length":"15"}]},{"pgid":"2.2","version":"70'2163","reported_seq":4318,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-03T17:47:24.086854+0000","last_change":"2023-10-03T17:40:08.675119+0000","last_active":"2023-10-03T17:47:24.086854+0000","last_peered":"2023-10-03T17:47:24.086854+0000","last_clean":"2023-10-03T17:47:24.086854+0000","last_became_active":"2023-10-03T17:34:42.270625+0000","last_became_peered":"2023-10-03T17:34:42.270625+0000","last_unstale":"2023-10-03T17:47:24.086854+0000","last_undegraded":"2023-10-03T17:47:24.086854+0000","last_fullsized":"2023-10-03T17:47:24.086854+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'841","last_scrub_stamp":"2023-10-03T17:40:08.674910+0000","last_deep_scrub":"46'841","last_deep_scrub_stamp":"2023-10-03T17:40:08.674910+0000","last_clean_scrub_stamp":"2023-10-03T17:40:08.674910+0000","objects_scrubbed":11,"log_size":2163,"log_dups_size":0,"ondisk_log_size":2163,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T21:05:42.790614+0000","scrub_duration":0.039189555000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":959,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":6042,"num_read_kb":4994,"num_write":229,"num_write_kb":197,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2564,"num_omap_keys":60,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"70'82919","reported_seq":146518,"reported_epoch":70,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T17:47:21.357353+0000","last_change":"2023-10-03T17:43:28.494717+0000","last_active":"2023-10-03T17:47:21.357353+0000","last_peered":"2023-10-03T17:47:21.357353+0000","last_clean":"2023-10-03T17:47:21.357353+0000","last_became_active":"2023-10-03T17:34:48.310555+0000","last_became_peered":"2023-10-03T17:34:48.310555+0000","last_unstale":"2023-10-03T17:47:21.357353+0000","last_undegraded":"2023-10-03T17:47:21.357353+0000","last_fullsized":"2023-10-03T17:47:21.357353+0000","mapping_epoch":16,"log_start":"61'72876","ondisk_log_start":"61'72876","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_clean_scrub_stamp":"2023-10-03T17:34:47.281095+0000","objects_scrubbed":3601,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":60,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 232s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086289699999999995,"stat_sum":{"num_bytes":5258219520,"num_objects":9411,"num_object_clones":8169,"num_object_copies":18822,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9411,"num_whiteouts":0,"num_read":71387,"num_read_kb":266660,"num_write":71787,"num_write_kb":287148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"70'155","reported_seq":1759,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-03T17:47:19.081252+0000","last_change":"2023-10-03T17:40:17.641493+0000","last_active":"2023-10-03T17:47:19.081252+0000","last_peered":"2023-10-03T17:47:19.081252+0000","last_clean":"2023-10-03T17:47:19.081252+0000","last_became_active":"2023-10-03T17:34:42.278196+0000","last_became_peered":"2023-10-03T17:34:42.278196+0000","last_unstale":"2023-10-03T17:47:19.081252+0000","last_undegraded":"2023-10-03T17:47:19.081252+0000","last_fullsized":"2023-10-03T17:47:19.081252+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'30","last_scrub_stamp":"2023-10-03T17:40:17.640969+0000","last_deep_scrub":"46'30","last_deep_scrub_stamp":"2023-10-03T17:40:17.640969+0000","last_clean_scrub_stamp":"2023-10-03T17:40:17.640969+0000","objects_scrubbed":6,"log_size":155,"log_dups_size":0,"ondisk_log_size":155,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:04:02.954896+0000","scrub_duration":0.071054565,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":470,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":4496,"num_read_kb":3713,"num_write":147,"num_write_kb":125,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":903,"num_omap_keys":30,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"70'89593","reported_seq":153817,"reported_epoch":70,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T17:47:26.028397+0000","last_change":"2023-10-03T17:47:25.860867+0000","last_active":"2023-10-03T17:47:26.028397+0000","last_peered":"2023-10-03T17:47:26.028397+0000","last_clean":"2023-10-03T17:47:26.028397+0000","last_became_active":"2023-10-03T17:34:48.310912+0000","last_became_peered":"2023-10-03T17:34:48.310912+0000","last_unstale":"2023-10-03T17:47:26.028397+0000","last_undegraded":"2023-10-03T17:47:26.028397+0000","last_fullsized":"2023-10-03T17:47:26.028397+0000","mapping_epoch":16,"log_start":"66'79576","ondisk_log_start":"66'79576","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'58196","last_scrub_stamp":"2023-10-03T17:43:26.193075+0000","last_deep_scrub":"53'58196","last_deep_scrub_stamp":"2023-10-03T17:43:26.193075+0000","last_clean_scrub_stamp":"2023-10-03T17:43:26.193075+0000","objects_scrubbed":2326,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":13,"last_scrub_duration":196,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:57:33.401519+0000","scrub_duration":195.44739730800001,"objects_trimmed":400,"snaptrim_duration":22.479920843999999,"stat_sum":{"num_bytes":5383634944,"num_objects":6819,"num_object_clones":5548,"num_object_copies":13638,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6819,"num_whiteouts":0,"num_read":73627,"num_read_kb":276664,"num_write":74245,"num_write_kb":296980,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"54","length":"1e"},{"start":"7c","length":"8"},{"start":"8a","length":"2"},{"start":"90","length":"7"}]},{"pgid":"2.0","version":"69'81","reported_seq":692,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-03T17:47:14.626129+0000","last_change":"2023-10-03T17:40:10.708344+0000","last_active":"2023-10-03T17:47:14.626129+0000","last_peered":"2023-10-03T17:47:14.626129+0000","last_clean":"2023-10-03T17:47:14.626129+0000","last_became_active":"2023-10-03T17:34:42.272306+0000","last_became_peered":"2023-10-03T17:34:42.272306+0000","last_unstale":"2023-10-03T17:47:14.626129+0000","last_undegraded":"2023-10-03T17:47:14.626129+0000","last_fullsized":"2023-10-03T17:47:14.626129+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'20","last_scrub_stamp":"2023-10-03T17:40:10.708060+0000","last_deep_scrub":"46'20","last_deep_scrub_stamp":"2023-10-03T17:40:10.708060+0000","last_clean_scrub_stamp":"2023-10-03T17:40:10.708060+0000","objects_scrubbed":7,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:03:09.097252+0000","scrub_duration":0.042279491000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":855,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1538,"num_read_kb":1253,"num_write":74,"num_write_kb":62,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":301,"num_omap_keys":10,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"70'89485","reported_seq":154081,"reported_epoch":70,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T17:47:21.379794+0000","last_change":"2023-10-03T17:47:19.301927+0000","last_active":"2023-10-03T17:47:21.379794+0000","last_peered":"2023-10-03T17:47:21.379794+0000","last_clean":"2023-10-03T17:47:21.379794+0000","last_became_active":"2023-10-03T17:34:48.310612+0000","last_became_peered":"2023-10-03T17:34:48.310612+0000","last_unstale":"2023-10-03T17:47:21.379794+0000","last_undegraded":"2023-10-03T17:47:21.379794+0000","last_fullsized":"2023-10-03T17:47:21.379794+0000","mapping_epoch":16,"log_start":"66'79448","ondisk_log_start":"66'79448","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'75807","last_scrub_stamp":"2023-10-03T17:45:43.833310+0000","last_deep_scrub":"61'75807","last_deep_scrub_stamp":"2023-10-03T17:45:43.833310+0000","last_clean_scrub_stamp":"2023-10-03T17:45:43.833310+0000","objects_scrubbed":3607,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":29,"last_scrub_duration":315,"scrub_schedule":"queued for deep scrub","scrub_duration":314.26374841099999,"objects_trimmed":317,"snaptrim_duration":15.950474407,"stat_sum":{"num_bytes":5447622675,"num_objects":7783,"num_object_clones":6495,"num_object_copies":15566,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7783,"num_whiteouts":0,"num_read":73525,"num_read_kb":275556,"num_write":74789,"num_write_kb":299150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"54","length":"1e"},{"start":"7c","length":"1"}]},{"pgid":"2.3","version":"69'38","reported_seq":145,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-03T17:47:03.351093+0000","last_change":"2023-10-03T17:40:24.472734+0000","last_active":"2023-10-03T17:47:03.351093+0000","last_peered":"2023-10-03T17:47:03.351093+0000","last_clean":"2023-10-03T17:47:03.351093+0000","last_became_active":"2023-10-03T17:34:42.270194+0000","last_became_peered":"2023-10-03T17:34:42.270194+0000","last_unstale":"2023-10-03T17:47:03.351093+0000","last_undegraded":"2023-10-03T17:47:03.351093+0000","last_fullsized":"2023-10-03T17:47:03.351093+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'8","last_scrub_stamp":"2023-10-03T17:40:24.472573+0000","last_deep_scrub":"46'8","last_deep_scrub_stamp":"2023-10-03T17:40:24.472573+0000","last_clean_scrub_stamp":"2023-10-03T17:40:24.472573+0000","objects_scrubbed":4,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T18:28:04.735033+0000","scrub_duration":0.024990637999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":38,"num_read_kb":20,"num_write":34,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"46'99","reported_seq":213,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-03T17:47:02.356594+0000","last_change":"2023-10-03T17:40:10.452341+0000","last_active":"2023-10-03T17:47:02.356594+0000","last_peered":"2023-10-03T17:47:02.356594+0000","last_clean":"2023-10-03T17:47:02.356594+0000","last_became_active":"2023-10-03T17:34:39.502393+0000","last_became_peered":"2023-10-03T17:34:39.502393+0000","last_unstale":"2023-10-03T17:47:02.356594+0000","last_undegraded":"2023-10-03T17:47:02.356594+0000","last_fullsized":"2023-10-03T17:47:02.356594+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'99","last_scrub_stamp":"2023-10-03T17:40:10.449099+0000","last_deep_scrub":"46'99","last_deep_scrub_stamp":"2023-10-03T17:40:10.449099+0000","last_clean_scrub_stamp":"2023-10-03T17:40:10.449099+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T20:18:04.958666+0000","scrub_duration":0.043225012,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":21679194131,"num_objects":32717,"num_object_clones":27596,"num_object_copies":65434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32717,"num_whiteouts":0,"num_read":294674,"num_read_kb":1104008,"num_write":298088,"num_write_kb":1192346,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2438746112,"data_stored":12184858662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40160,"ondisk_log_size":40160,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":5372,"num_objects":143,"num_object_clones":0,"num_object_copies":286,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":143,"num_whiteouts":0,"num_read":32383,"num_read_kb":27032,"num_write":1183,"num_write_kb":1003,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":983040,"data_stored":10744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8804,"ondisk_log_size":8804,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":7460,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738527,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2912156,"kb_used_data":1189716,"kb_used_omap":1,"kb_used_meta":1722430,"kb_avail":91459684,"statfs":{"total":96636764160,"available":93654716416,"internally_reserved":0,"allocated":1218269184,"data_stored":6082440196,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1763768774},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,1,4,6,10],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 17:47:00 2023","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.436},"min":{"1min":0.17199999999999999,"5min":0.16500000000000001,"15min":0.14299999999999999},"max":{"1min":3.6539999999999999,"5min":4.1539999999999999,"15min":4.4640000000000004},"last":3.879},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.42399999999999999,"15min":0.44500000000000001},"min":{"1min":0.17999999999999999,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":1.758,"5min":4.4180000000000001,"15min":4.4180000000000001},"last":0.38200000000000001}]},{"osd":1,"last update":"Tue Oct 3 17:47:00 2023","interfaces":[{"interface":"back","average":{"1min":0.32400000000000001,"5min":0.61899999999999999,"15min":0.51200000000000001},"min":{"1min":0.221,"5min":0.184,"15min":0.16600000000000001},"max":{"1min":0.66600000000000004,"5min":8.1059999999999999,"15min":8.1059999999999999},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.52900000000000003,"15min":0.48399999999999999},"min":{"1min":0.223,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":1.627,"5min":4.1120000000000001,"15min":4.1120000000000001},"last":0.34599999999999997}]}]},{"osd":1,"up_from":8,"seq":34359738525,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1477484,"kb_used_data":587044,"kb_used_omap":1,"kb_used_meta":890430,"kb_avail":92894356,"statfs":{"total":96636764160,"available":95123820544,"internally_reserved":0,"allocated":601133056,"data_stored":3201072191,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":911800774},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,4,4,0,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 17:47:09 2023","interfaces":[{"interface":"back","average":{"1min":0.26700000000000002,"5min":0.30599999999999999,"15min":0.38400000000000001},"min":{"1min":0.20599999999999999,"5min":0.17199999999999999,"15min":0.161},"max":{"1min":0.46400000000000002,"5min":2.3809999999999998,"15min":4.1879999999999997},"last":0.20799999999999999},{"interface":"front","average":{"1min":0.28999999999999998,"5min":0.43099999999999999,"15min":0.42699999999999999},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.52900000000000003,"5min":4.5759999999999996,"15min":4.5759999999999996},"last":0.34100000000000003}]},{"osd":2,"last update":"Tue Oct 3 17:47:09 2023","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.41199999999999998,"15min":0.42899999999999999},"min":{"1min":0.19800000000000001,"5min":0.187,"15min":0.159},"max":{"1min":2.613,"5min":2.613,"15min":3.0390000000000001},"last":0.247},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.376,"15min":0.42499999999999999},"min":{"1min":0.23499999999999999,"5min":0.16400000000000001,"15min":0.16400000000000001},"max":{"1min":2.6970000000000001,"5min":2.6970000000000001,"15min":3.0699999999999998},"last":0.38500000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738526,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1477604,"kb_used_data":606556,"kb_used_omap":1,"kb_used_meta":871038,"kb_avail":92894236,"statfs":{"total":96636764160,"available":95123697664,"internally_reserved":0,"allocated":621113344,"data_stored":2892850507,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":891943366},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,2,2,9,10],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 17:46:57 2023","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.38800000000000001,"15min":0.42799999999999999},"min":{"1min":0.182,"5min":0.17100000000000001,"15min":0.154},"max":{"1min":3.4649999999999999,"5min":3.4649999999999999,"15min":5.7649999999999997},"last":0.251},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.54200000000000004,"15min":0.47099999999999997},"min":{"1min":0.16900000000000001,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":3.4279999999999999,"5min":9.4770000000000003,"15min":9.4770000000000003},"last":0.318}]},{"osd":2,"last update":"Tue Oct 3 17:46:57 2023","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.66400000000000003,"15min":0.54100000000000004},"min":{"1min":0.20000000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":3.4540000000000002,"5min":9.4149999999999991,"15min":9.4149999999999991},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.61499999999999999,"15min":0.52100000000000002},"min":{"1min":0.253,"5min":0.189,"15min":0.189},"max":{"1min":3.4249999999999998,"5min":5.218,"15min":5.218},"last":0.30299999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":3611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":5372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":159744,"data_stored":1761,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":620593152,"data_stored":2894426112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":600379392,"data_stored":3206348819,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1217773568,"data_stored":6084083731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-03T17:47:26.735 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-03T17:34:47.281095+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=34, tm_sec=47, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=40, tm_sec=6, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T17:47:26.736 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T17:47:29.999 INFO:tasks.ceph.mgr.x.smithi111.stderr:2023-10-03T17:47:29.998+0000 7fe48bb97640 -1 calc_pg_upmaps abort due to max <= 0 2023-10-03T17:47:46.737 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T17:47:47.348 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-03T17:47:47.349 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-10-03T17:47:47.377 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":463,"stamp":"2023-10-03T17:47:46.129002+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":21709932335,"num_objects":32802,"num_object_clones":27536,"num_object_copies":65604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32802,"num_whiteouts":0,"num_read":332632,"num_read_kb":1152615,"num_write":305102,"num_write_kb":1218019,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":49164,"ondisk_log_size":49164,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":6009876,"kb_used_data":2431612,"kb_used_omap":4,"kb_used_meta":3578235,"kb_avail":277105644,"statfs":{"total":289910292480,"available":283756179456,"internally_reserved":0,"allocated":2489970688,"data_stored":12276661646,"data_compressed":7460,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4782,"internal_metadata":3664112978},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,4,1,12,16,19,15],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":17489920,"num_objects":-205,"num_object_clones":-205,"num_object_copies":-410,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-205,"num_whiteouts":0,"num_read":3229,"num_read_kb":12696,"num_write":3333,"num_write_kb":13332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002844"},"pg_stats":[{"pgid":"2.7","version":"70'1596","reported_seq":3222,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-03T17:47:39.446065+0000","last_change":"2023-10-03T17:40:17.407011+0000","last_active":"2023-10-03T17:47:39.446065+0000","last_peered":"2023-10-03T17:47:39.446065+0000","last_clean":"2023-10-03T17:47:39.446065+0000","last_became_active":"2023-10-03T17:34:42.276624+0000","last_became_peered":"2023-10-03T17:34:42.276624+0000","last_unstale":"2023-10-03T17:47:39.446065+0000","last_undegraded":"2023-10-03T17:47:39.446065+0000","last_fullsized":"2023-10-03T17:47:39.446065+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'673","last_scrub_stamp":"2023-10-03T17:40:17.406813+0000","last_deep_scrub":"46'673","last_deep_scrub_stamp":"2023-10-03T17:40:17.406813+0000","last_clean_scrub_stamp":"2023-10-03T17:40:17.406813+0000","objects_scrubbed":7,"log_size":1596,"log_dups_size":0,"ondisk_log_size":1596,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T21:38:35.056889+0000","scrub_duration":0.013778076,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":739,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":4514,"num_read_kb":3722,"num_write":161,"num_write_kb":135,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":903,"num_omap_keys":30,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"70'2666","reported_seq":6419,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-03T17:47:39.446739+0000","last_change":"2023-10-03T17:40:13.376519+0000","last_active":"2023-10-03T17:47:39.446739+0000","last_peered":"2023-10-03T17:47:39.446739+0000","last_clean":"2023-10-03T17:47:39.446739+0000","last_became_active":"2023-10-03T17:34:42.271201+0000","last_became_peered":"2023-10-03T17:34:42.271201+0000","last_unstale":"2023-10-03T17:47:39.446739+0000","last_undegraded":"2023-10-03T17:47:39.446739+0000","last_fullsized":"2023-10-03T17:47:39.446739+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'1129","last_scrub_stamp":"2023-10-03T17:40:13.376332+0000","last_deep_scrub":"46'1129","last_deep_scrub_stamp":"2023-10-03T17:40:13.376332+0000","last_clean_scrub_stamp":"2023-10-03T17:40:13.376332+0000","objects_scrubbed":11,"log_size":2666,"log_dups_size":0,"ondisk_log_size":2666,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T18:59:17.088585+0000","scrub_duration":0.015998675,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":723,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":9082,"num_read_kb":7772,"num_write":297,"num_write_kb":255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2912,"num_omap_keys":93,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"69'36","reported_seq":142,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-03T17:47:03.351077+0000","last_change":"2023-10-03T17:40:25.496010+0000","last_active":"2023-10-03T17:47:03.351077+0000","last_peered":"2023-10-03T17:47:03.351077+0000","last_clean":"2023-10-03T17:47:03.351077+0000","last_became_active":"2023-10-03T17:34:42.276941+0000","last_became_peered":"2023-10-03T17:34:42.276941+0000","last_unstale":"2023-10-03T17:47:03.351077+0000","last_undegraded":"2023-10-03T17:47:03.351077+0000","last_fullsized":"2023-10-03T17:47:03.351077+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'14","last_scrub_stamp":"2023-10-03T17:40:25.495827+0000","last_deep_scrub":"46'14","last_deep_scrub_stamp":"2023-10-03T17:40:25.495827+0000","last_clean_scrub_stamp":"2023-10-03T17:40:25.495827+0000","objects_scrubbed":7,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:05:44.939663+0000","scrub_duration":0.0133078,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":957,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":43,"num_read_kb":16,"num_write":29,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"70'2129","reported_seq":4852,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-03T17:47:39.446351+0000","last_change":"2023-10-03T17:40:26.513902+0000","last_active":"2023-10-03T17:47:39.446351+0000","last_peered":"2023-10-03T17:47:39.446351+0000","last_clean":"2023-10-03T17:47:39.446351+0000","last_became_active":"2023-10-03T17:34:42.276459+0000","last_became_peered":"2023-10-03T17:34:42.276459+0000","last_unstale":"2023-10-03T17:47:39.446351+0000","last_undegraded":"2023-10-03T17:47:39.446351+0000","last_fullsized":"2023-10-03T17:47:39.446351+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'964","last_scrub_stamp":"2023-10-03T17:40:26.513729+0000","last_deep_scrub":"46'964","last_deep_scrub_stamp":"2023-10-03T17:40:26.513729+0000","last_clean_scrub_stamp":"2023-10-03T17:40:26.513729+0000","objects_scrubbed":10,"log_size":2129,"log_dups_size":0,"ondisk_log_size":2129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:41:33.182751+0000","scrub_duration":0.018190280999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":6630,"num_read_kb":5542,"num_write":212,"num_write_kb":182,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1731,"num_omap_keys":80,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"70'92937","reported_seq":161055,"reported_epoch":70,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T17:47:41.190694+0000","last_change":"2023-10-03T17:47:40.874430+0000","last_active":"2023-10-03T17:47:41.190694+0000","last_peered":"2023-10-03T17:47:41.190694+0000","last_clean":"2023-10-03T17:47:41.190694+0000","last_became_active":"2023-10-03T17:34:48.310701+0000","last_became_peered":"2023-10-03T17:34:48.310701+0000","last_unstale":"2023-10-03T17:47:41.190694+0000","last_undegraded":"2023-10-03T17:47:41.190694+0000","last_fullsized":"2023-10-03T17:47:41.190694+0000","mapping_epoch":16,"log_start":"66'82935","ondisk_log_start":"66'82935","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'80695","last_scrub_stamp":"2023-10-03T17:46:13.982571+0000","last_deep_scrub":"66'80695","last_deep_scrub_stamp":"2023-10-03T17:46:13.982571+0000","last_clean_scrub_stamp":"2023-10-03T17:46:13.982571+0000","objects_scrubbed":3410,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":34,"last_scrub_duration":363,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T21:31:03.424497+0000","scrub_duration":362.26574615599998,"objects_trimmed":648,"snaptrim_duration":37.513080316,"stat_sum":{"num_bytes":5596344320,"num_objects":8590,"num_object_clones":7270,"num_object_copies":17180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8590,"num_whiteouts":0,"num_read":77402,"num_read_kb":290108,"num_write":78527,"num_write_kb":314108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"54","length":"1a"}]},{"pgid":"2.2","version":"70'2183","reported_seq":4350,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-03T17:47:39.446150+0000","last_change":"2023-10-03T17:40:08.675119+0000","last_active":"2023-10-03T17:47:39.446150+0000","last_peered":"2023-10-03T17:47:39.446150+0000","last_clean":"2023-10-03T17:47:39.446150+0000","last_became_active":"2023-10-03T17:34:42.270625+0000","last_became_peered":"2023-10-03T17:34:42.270625+0000","last_unstale":"2023-10-03T17:47:39.446150+0000","last_undegraded":"2023-10-03T17:47:39.446150+0000","last_fullsized":"2023-10-03T17:47:39.446150+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'841","last_scrub_stamp":"2023-10-03T17:40:08.674910+0000","last_deep_scrub":"46'841","last_deep_scrub_stamp":"2023-10-03T17:40:08.674910+0000","last_clean_scrub_stamp":"2023-10-03T17:40:08.674910+0000","objects_scrubbed":11,"log_size":2183,"log_dups_size":0,"ondisk_log_size":2183,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T21:05:42.790614+0000","scrub_duration":0.039189555000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":959,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":6042,"num_read_kb":4994,"num_write":229,"num_write_kb":197,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2564,"num_omap_keys":60,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"70'85071","reported_seq":149891,"reported_epoch":70,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T17:47:41.525900+0000","last_change":"2023-10-03T17:43:28.494717+0000","last_active":"2023-10-03T17:47:41.525900+0000","last_peered":"2023-10-03T17:47:41.525900+0000","last_clean":"2023-10-03T17:47:41.525900+0000","last_became_active":"2023-10-03T17:34:48.310555+0000","last_became_peered":"2023-10-03T17:34:48.310555+0000","last_unstale":"2023-10-03T17:47:41.525900+0000","last_undegraded":"2023-10-03T17:47:41.525900+0000","last_fullsized":"2023-10-03T17:47:41.525900+0000","mapping_epoch":16,"log_start":"66'74978","ondisk_log_start":"66'74978","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_clean_scrub_stamp":"2023-10-03T17:34:47.281095+0000","objects_scrubbed":3950,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":60,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 253s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086289699999999995,"stat_sum":{"num_bytes":5266661376,"num_objects":9695,"num_object_clones":8453,"num_object_copies":19390,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9695,"num_whiteouts":0,"num_read":72873,"num_read_kb":272484,"num_write":73346,"num_write_kb":293384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"70'155","reported_seq":1771,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-03T17:47:39.451428+0000","last_change":"2023-10-03T17:40:17.641493+0000","last_active":"2023-10-03T17:47:39.451428+0000","last_peered":"2023-10-03T17:47:39.451428+0000","last_clean":"2023-10-03T17:47:39.451428+0000","last_became_active":"2023-10-03T17:34:42.278196+0000","last_became_peered":"2023-10-03T17:34:42.278196+0000","last_unstale":"2023-10-03T17:47:39.451428+0000","last_undegraded":"2023-10-03T17:47:39.451428+0000","last_fullsized":"2023-10-03T17:47:39.451428+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'30","last_scrub_stamp":"2023-10-03T17:40:17.640969+0000","last_deep_scrub":"46'30","last_deep_scrub_stamp":"2023-10-03T17:40:17.640969+0000","last_clean_scrub_stamp":"2023-10-03T17:40:17.640969+0000","objects_scrubbed":6,"log_size":155,"log_dups_size":0,"ondisk_log_size":155,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:04:02.954896+0000","scrub_duration":0.071054565,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":470,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":4496,"num_read_kb":3713,"num_write":147,"num_write_kb":125,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":903,"num_omap_keys":30,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"70'91654","reported_seq":156742,"reported_epoch":70,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T17:47:41.169601+0000","last_change":"2023-10-03T17:47:39.945575+0000","last_active":"2023-10-03T17:47:41.169601+0000","last_peered":"2023-10-03T17:47:41.169601+0000","last_clean":"2023-10-03T17:47:41.169601+0000","last_became_active":"2023-10-03T17:34:48.310912+0000","last_became_peered":"2023-10-03T17:34:48.310912+0000","last_unstale":"2023-10-03T17:47:41.169601+0000","last_undegraded":"2023-10-03T17:47:41.169601+0000","last_fullsized":"2023-10-03T17:47:41.169601+0000","mapping_epoch":16,"log_start":"66'81581","ondisk_log_start":"66'81581","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'58196","last_scrub_stamp":"2023-10-03T17:43:26.193075+0000","last_deep_scrub":"53'58196","last_deep_scrub_stamp":"2023-10-03T17:43:26.193075+0000","last_clean_scrub_stamp":"2023-10-03T17:43:26.193075+0000","objects_scrubbed":2326,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":8,"last_scrub_duration":196,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:57:33.401519+0000","scrub_duration":195.44739730800001,"objects_trimmed":730,"snaptrim_duration":36.565750084999998,"stat_sum":{"num_bytes":5389447168,"num_objects":6660,"num_object_clones":5389,"num_object_copies":13320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6660,"num_whiteouts":0,"num_read":74800,"num_read_kb":281280,"num_write":75471,"num_write_kb":301884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"54","length":"1e"},{"start":"7c","length":"8"},{"start":"8a","length":"2"},{"start":"90","length":"a"},{"start":"a4","length":"2"}]},{"pgid":"2.0","version":"69'81","reported_seq":696,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-03T17:47:34.328497+0000","last_change":"2023-10-03T17:40:10.708344+0000","last_active":"2023-10-03T17:47:34.328497+0000","last_peered":"2023-10-03T17:47:34.328497+0000","last_clean":"2023-10-03T17:47:34.328497+0000","last_became_active":"2023-10-03T17:34:42.272306+0000","last_became_peered":"2023-10-03T17:34:42.272306+0000","last_unstale":"2023-10-03T17:47:34.328497+0000","last_undegraded":"2023-10-03T17:47:34.328497+0000","last_fullsized":"2023-10-03T17:47:34.328497+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'20","last_scrub_stamp":"2023-10-03T17:40:10.708060+0000","last_deep_scrub":"46'20","last_deep_scrub_stamp":"2023-10-03T17:40:10.708060+0000","last_clean_scrub_stamp":"2023-10-03T17:40:10.708060+0000","objects_scrubbed":7,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:03:09.097252+0000","scrub_duration":0.042279491000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":855,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1538,"num_read_kb":1253,"num_write":74,"num_write_kb":62,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":301,"num_omap_keys":10,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"70'92173","reported_seq":157864,"reported_epoch":70,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T17:47:41.527046+0000","last_change":"2023-10-03T17:47:39.883114+0000","last_active":"2023-10-03T17:47:41.527046+0000","last_peered":"2023-10-03T17:47:41.527046+0000","last_clean":"2023-10-03T17:47:41.527046+0000","last_became_active":"2023-10-03T17:34:48.310612+0000","last_became_peered":"2023-10-03T17:34:48.310612+0000","last_unstale":"2023-10-03T17:47:41.527046+0000","last_undegraded":"2023-10-03T17:47:41.527046+0000","last_fullsized":"2023-10-03T17:47:41.527046+0000","mapping_epoch":16,"log_start":"66'82160","ondisk_log_start":"66'82160","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'75807","last_scrub_stamp":"2023-10-03T17:45:43.833310+0000","last_deep_scrub":"61'75807","last_deep_scrub_stamp":"2023-10-03T17:45:43.833310+0000","last_clean_scrub_stamp":"2023-10-03T17:45:43.833310+0000","objects_scrubbed":3607,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":23,"last_scrub_duration":315,"scrub_schedule":"queued for deep scrub","scrub_duration":314.26374841099999,"objects_trimmed":687,"snaptrim_duration":36.530733570999999,"stat_sum":{"num_bytes":5456883731,"num_objects":7712,"num_object_clones":6424,"num_object_copies":15424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7712,"num_whiteouts":0,"num_read":75068,"num_read_kb":281620,"num_write":76438,"num_write_kb":305746,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"54","length":"1e"},{"start":"7c","length":"7"}]},{"pgid":"2.3","version":"69'38","reported_seq":145,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-03T17:47:03.351093+0000","last_change":"2023-10-03T17:40:24.472734+0000","last_active":"2023-10-03T17:47:03.351093+0000","last_peered":"2023-10-03T17:47:03.351093+0000","last_clean":"2023-10-03T17:47:03.351093+0000","last_became_active":"2023-10-03T17:34:42.270194+0000","last_became_peered":"2023-10-03T17:34:42.270194+0000","last_unstale":"2023-10-03T17:47:03.351093+0000","last_undegraded":"2023-10-03T17:47:03.351093+0000","last_fullsized":"2023-10-03T17:47:03.351093+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'8","last_scrub_stamp":"2023-10-03T17:40:24.472573+0000","last_deep_scrub":"46'8","last_deep_scrub_stamp":"2023-10-03T17:40:24.472573+0000","last_clean_scrub_stamp":"2023-10-03T17:40:24.472573+0000","objects_scrubbed":4,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T18:28:04.735033+0000","scrub_duration":0.024990637999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":38,"num_read_kb":20,"num_write":34,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"46'99","reported_seq":213,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-03T17:47:02.356594+0000","last_change":"2023-10-03T17:40:10.452341+0000","last_active":"2023-10-03T17:47:02.356594+0000","last_peered":"2023-10-03T17:47:02.356594+0000","last_clean":"2023-10-03T17:47:02.356594+0000","last_became_active":"2023-10-03T17:34:39.502393+0000","last_became_peered":"2023-10-03T17:34:39.502393+0000","last_unstale":"2023-10-03T17:47:02.356594+0000","last_undegraded":"2023-10-03T17:47:02.356594+0000","last_fullsized":"2023-10-03T17:47:02.356594+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'99","last_scrub_stamp":"2023-10-03T17:40:10.449099+0000","last_deep_scrub":"46'99","last_deep_scrub_stamp":"2023-10-03T17:40:10.449099+0000","last_clean_scrub_stamp":"2023-10-03T17:40:10.449099+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T20:18:04.958666+0000","scrub_duration":0.043225012,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":21709336595,"num_objects":32657,"num_object_clones":27536,"num_object_copies":65314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32657,"num_whiteouts":0,"num_read":300143,"num_read_kb":1125492,"num_write":303782,"num_write_kb":1215122,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2488582144,"data_stored":12271530022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40181,"ondisk_log_size":40181,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":5372,"num_objects":143,"num_object_clones":0,"num_object_copies":286,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":143,"num_whiteouts":0,"num_read":32383,"num_read_kb":27032,"num_write":1183,"num_write_kb":1003,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":983040,"data_stored":10744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8884,"ondisk_log_size":8884,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":7460,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738531,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2982268,"kb_used_data":1215476,"kb_used_omap":1,"kb_used_meta":1766782,"kb_avail":91389572,"statfs":{"total":96636764160,"available":93582921728,"internally_reserved":0,"allocated":1244647424,"data_stored":6136036356,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1809185222},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,1,7,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 17:47:00 2023","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.436},"min":{"1min":0.17199999999999999,"5min":0.16500000000000001,"15min":0.14299999999999999},"max":{"1min":3.6539999999999999,"5min":4.1539999999999999,"15min":4.4640000000000004},"last":0.73399999999999999},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.42399999999999999,"15min":0.44500000000000001},"min":{"1min":0.17999999999999999,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":1.758,"5min":4.4180000000000001,"15min":4.4180000000000001},"last":2.9049999999999998}]},{"osd":1,"last update":"Tue Oct 3 17:47:00 2023","interfaces":[{"interface":"back","average":{"1min":0.32400000000000001,"5min":0.61899999999999999,"15min":0.51200000000000001},"min":{"1min":0.221,"5min":0.184,"15min":0.16600000000000001},"max":{"1min":0.66600000000000004,"5min":8.1059999999999999,"15min":8.1059999999999999},"last":0.35499999999999998},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.52900000000000003,"15min":0.48399999999999999},"min":{"1min":0.223,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":1.627,"5min":4.1120000000000001,"15min":4.1120000000000001},"last":0.33300000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738529,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1526212,"kb_used_data":599676,"kb_used_omap":1,"kb_used_meta":926526,"kb_avail":92845628,"statfs":{"total":96636764160,"available":95073923072,"internally_reserved":0,"allocated":614068224,"data_stored":3280452671,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":948763078},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,1,9,5,10],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 17:47:09 2023","interfaces":[{"interface":"back","average":{"1min":0.26700000000000002,"5min":0.30599999999999999,"15min":0.38400000000000001},"min":{"1min":0.20599999999999999,"5min":0.17199999999999999,"15min":0.161},"max":{"1min":0.46400000000000002,"5min":2.3809999999999998,"15min":4.1879999999999997},"last":0.23300000000000001},{"interface":"front","average":{"1min":0.28999999999999998,"5min":0.43099999999999999,"15min":0.42699999999999999},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.52900000000000003,"5min":4.5759999999999996,"15min":4.5759999999999996},"last":0.434}]},{"osd":2,"last update":"Tue Oct 3 17:47:09 2023","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.41199999999999998,"15min":0.42899999999999999},"min":{"1min":0.19800000000000001,"5min":0.187,"15min":0.159},"max":{"1min":2.613,"5min":2.613,"15min":3.0390000000000001},"last":0.21099999999999999},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.376,"15min":0.42499999999999999},"min":{"1min":0.23499999999999999,"5min":0.16400000000000001,"15min":0.16400000000000001},"max":{"1min":2.6970000000000001,"5min":2.6970000000000001,"15min":3.0699999999999998},"last":0.30299999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738529,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1501396,"kb_used_data":616460,"kb_used_omap":1,"kb_used_meta":884926,"kb_avail":92870444,"statfs":{"total":96636764160,"available":95099334656,"internally_reserved":0,"allocated":631255040,"data_stored":2860172619,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":906164678},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,1,0,10,0,8,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 17:46:57 2023","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.38800000000000001,"15min":0.42799999999999999},"min":{"1min":0.182,"5min":0.17100000000000001,"15min":0.154},"max":{"1min":3.4649999999999999,"5min":3.4649999999999999,"15min":5.7649999999999997},"last":0.188},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.54200000000000004,"15min":0.47099999999999997},"min":{"1min":0.16900000000000001,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":3.4279999999999999,"5min":9.4770000000000003,"15min":9.4770000000000003},"last":0.27100000000000002}]},{"osd":2,"last update":"Tue Oct 3 17:46:57 2023","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.66400000000000003,"15min":0.54100000000000004},"min":{"1min":0.20000000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":3.4540000000000002,"5min":9.4149999999999991,"15min":9.4149999999999991},"last":0.312},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.61499999999999999,"15min":0.52100000000000002},"min":{"1min":0.253,"5min":0.189,"15min":0.189},"max":{"1min":3.4249999999999998,"5min":5.218,"15min":5.218},"last":0.29199999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":3611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":5372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":159744,"data_stored":1761,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":630788096,"data_stored":2855305216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":613511168,"data_stored":3281465363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1244282880,"data_stored":6134759443,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-03T17:47:47.379 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-03T17:34:47.281095+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=34, tm_sec=47, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=40, tm_sec=6, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T17:47:47.379 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T17:48:01.454 INFO:tasks.ceph.osd.0.smithi111.stderr:2023-10-03T17:48:01.453+0000 7ffbac289640 -1 reset not still connected to 0x5563b44301a0 2023-10-03T17:48:01.455 INFO:tasks.ceph.osd.0.smithi111.stderr:2023-10-03T17:48:01.453+0000 7ffbac289640 -1 reset not still connected to 0x5563b6604820 2023-10-03T17:48:07.380 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T17:48:07.921 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-03T17:48:07.922 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-10-03T17:48:07.953 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":477,"stamp":"2023-10-03T17:48:06.136652+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":21745452847,"num_objects":31953,"num_object_clones":26687,"num_object_copies":63906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31953,"num_whiteouts":0,"num_read":341368,"num_read_kb":1180676,"num_write":311773,"num_write_kb":1244354,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":49309,"ondisk_log_size":49309,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":6184560,"kb_used_data":2484108,"kb_used_omap":4,"kb_used_meta":3700411,"kb_avail":276930960,"statfs":{"total":289910292480,"available":283577303040,"internally_reserved":0,"allocated":2543726592,"data_stored":11976138159,"data_compressed":7460,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4782,"internal_metadata":3789221202},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,5,9,23,20,17,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":8634368,"num_objects":-138,"num_object_clones":-138,"num_object_copies":-276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-138,"num_whiteouts":0,"num_read":3697,"num_read_kb":8257,"num_write":1689,"num_write_kb":6407,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.798553"},"pg_stats":[{"pgid":"2.7","version":"72'1626","reported_seq":3333,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-03T17:48:01.383590+0000","last_change":"2023-10-03T17:40:17.407011+0000","last_active":"2023-10-03T17:48:01.383590+0000","last_peered":"2023-10-03T17:48:01.383590+0000","last_clean":"2023-10-03T17:48:01.383590+0000","last_became_active":"2023-10-03T17:34:42.276624+0000","last_became_peered":"2023-10-03T17:34:42.276624+0000","last_unstale":"2023-10-03T17:48:01.383590+0000","last_undegraded":"2023-10-03T17:48:01.383590+0000","last_fullsized":"2023-10-03T17:48:01.383590+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'673","last_scrub_stamp":"2023-10-03T17:40:17.406813+0000","last_deep_scrub":"46'673","last_deep_scrub_stamp":"2023-10-03T17:40:17.406813+0000","last_clean_scrub_stamp":"2023-10-03T17:40:17.406813+0000","objects_scrubbed":7,"log_size":1626,"log_dups_size":0,"ondisk_log_size":1626,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T21:38:35.056889+0000","scrub_duration":0.013778076,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":839,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":4802,"num_read_kb":3962,"num_write":185,"num_write_kb":153,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":903,"num_omap_keys":30,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"72'2705","reported_seq":6652,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-03T17:48:01.501845+0000","last_change":"2023-10-03T17:40:13.376519+0000","last_active":"2023-10-03T17:48:01.501845+0000","last_peered":"2023-10-03T17:48:01.501845+0000","last_clean":"2023-10-03T17:48:01.501845+0000","last_became_active":"2023-10-03T17:34:42.271201+0000","last_became_peered":"2023-10-03T17:34:42.271201+0000","last_unstale":"2023-10-03T17:48:01.501845+0000","last_undegraded":"2023-10-03T17:48:01.501845+0000","last_fullsized":"2023-10-03T17:48:01.501845+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'1129","last_scrub_stamp":"2023-10-03T17:40:13.376332+0000","last_deep_scrub":"46'1129","last_deep_scrub_stamp":"2023-10-03T17:40:13.376332+0000","last_clean_scrub_stamp":"2023-10-03T17:40:13.376332+0000","objects_scrubbed":11,"log_size":2705,"log_dups_size":0,"ondisk_log_size":2705,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T18:59:17.088585+0000","scrub_duration":0.015998675,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":9620,"num_read_kb":8226,"num_write":319,"num_write_kb":270,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2912,"num_omap_keys":93,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"71'38","reported_seq":146,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-03T17:48:00.374555+0000","last_change":"2023-10-03T17:40:25.496010+0000","last_active":"2023-10-03T17:48:00.374555+0000","last_peered":"2023-10-03T17:48:00.374555+0000","last_clean":"2023-10-03T17:48:00.374555+0000","last_became_active":"2023-10-03T17:34:42.276941+0000","last_became_peered":"2023-10-03T17:34:42.276941+0000","last_unstale":"2023-10-03T17:48:00.374555+0000","last_undegraded":"2023-10-03T17:48:00.374555+0000","last_fullsized":"2023-10-03T17:48:00.374555+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'14","last_scrub_stamp":"2023-10-03T17:40:25.495827+0000","last_deep_scrub":"46'14","last_deep_scrub_stamp":"2023-10-03T17:40:25.495827+0000","last_clean_scrub_stamp":"2023-10-03T17:40:25.495827+0000","objects_scrubbed":7,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:05:44.939663+0000","scrub_duration":0.0133078,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1058,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":43,"num_read_kb":16,"num_write":31,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"72'2159","reported_seq":5017,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-03T17:48:01.504662+0000","last_change":"2023-10-03T17:40:26.513902+0000","last_active":"2023-10-03T17:48:01.504662+0000","last_peered":"2023-10-03T17:48:01.504662+0000","last_clean":"2023-10-03T17:48:01.504662+0000","last_became_active":"2023-10-03T17:34:42.276459+0000","last_became_peered":"2023-10-03T17:34:42.276459+0000","last_unstale":"2023-10-03T17:48:01.504662+0000","last_undegraded":"2023-10-03T17:48:01.504662+0000","last_fullsized":"2023-10-03T17:48:01.504662+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'964","last_scrub_stamp":"2023-10-03T17:40:26.513729+0000","last_deep_scrub":"46'964","last_deep_scrub_stamp":"2023-10-03T17:40:26.513729+0000","last_clean_scrub_stamp":"2023-10-03T17:40:26.513729+0000","objects_scrubbed":10,"log_size":2159,"log_dups_size":0,"ondisk_log_size":2159,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:41:33.182751+0000","scrub_duration":0.018190280999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":7063,"num_read_kb":5903,"num_write":230,"num_write_kb":196,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1731,"num_omap_keys":80,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"72'95657","reported_seq":165132,"reported_epoch":72,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T17:48:01.445265+0000","last_change":"2023-10-03T17:47:58.465560+0000","last_active":"2023-10-03T17:48:01.445265+0000","last_peered":"2023-10-03T17:48:01.445265+0000","last_clean":"2023-10-03T17:48:01.445265+0000","last_became_active":"2023-10-03T17:34:48.310701+0000","last_became_peered":"2023-10-03T17:34:48.310701+0000","last_unstale":"2023-10-03T17:48:01.445265+0000","last_undegraded":"2023-10-03T17:48:01.445265+0000","last_fullsized":"2023-10-03T17:48:01.445265+0000","mapping_epoch":16,"log_start":"66'85645","ondisk_log_start":"66'85645","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'80695","last_scrub_stamp":"2023-10-03T17:46:13.982571+0000","last_deep_scrub":"66'80695","last_deep_scrub_stamp":"2023-10-03T17:46:13.982571+0000","last_clean_scrub_stamp":"2023-10-03T17:46:13.982571+0000","objects_scrubbed":3410,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":38,"last_scrub_duration":363,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T18:39:24.473374+0000","scrub_duration":362.26574615599998,"objects_trimmed":2,"snaptrim_duration":55.104349055,"stat_sum":{"num_bytes":5604880384,"num_objects":8292,"num_object_clones":6972,"num_object_copies":16584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8292,"num_whiteouts":0,"num_read":79080,"num_read_kb":296712,"num_write":80246,"num_write_kb":320984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"54","length":"1e"},{"start":"7c","length":"2"}]},{"pgid":"2.2","version":"72'2254","reported_seq":4521,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-03T17:48:01.445008+0000","last_change":"2023-10-03T17:40:08.675119+0000","last_active":"2023-10-03T17:48:01.445008+0000","last_peered":"2023-10-03T17:48:01.445008+0000","last_clean":"2023-10-03T17:48:01.445008+0000","last_became_active":"2023-10-03T17:34:42.270625+0000","last_became_peered":"2023-10-03T17:34:42.270625+0000","last_unstale":"2023-10-03T17:48:01.445008+0000","last_undegraded":"2023-10-03T17:48:01.445008+0000","last_fullsized":"2023-10-03T17:48:01.445008+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'841","last_scrub_stamp":"2023-10-03T17:40:08.674910+0000","last_deep_scrub":"46'841","last_deep_scrub_stamp":"2023-10-03T17:40:08.674910+0000","last_clean_scrub_stamp":"2023-10-03T17:40:08.674910+0000","objects_scrubbed":11,"log_size":2254,"log_dups_size":0,"ondisk_log_size":2254,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T21:05:42.790614+0000","scrub_duration":0.039189555000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":959,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":6408,"num_read_kb":5299,"num_write":251,"num_write_kb":213,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2564,"num_omap_keys":60,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"72'87117","reported_seq":153431,"reported_epoch":72,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T17:48:01.686827+0000","last_change":"2023-10-03T17:43:28.494717+0000","last_active":"2023-10-03T17:48:01.686827+0000","last_peered":"2023-10-03T17:48:01.686827+0000","last_clean":"2023-10-03T17:48:01.686827+0000","last_became_active":"2023-10-03T17:34:48.310555+0000","last_became_peered":"2023-10-03T17:34:48.310555+0000","last_unstale":"2023-10-03T17:48:01.686827+0000","last_undegraded":"2023-10-03T17:48:01.686827+0000","last_fullsized":"2023-10-03T17:48:01.686827+0000","mapping_epoch":16,"log_start":"66'77080","ondisk_log_start":"66'77080","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_clean_scrub_stamp":"2023-10-03T17:34:47.281095+0000","objects_scrubbed":4367,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":70,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 273s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086289699999999995,"stat_sum":{"num_bytes":5274566656,"num_objects":9831,"num_object_clones":8589,"num_object_copies":19662,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9831,"num_whiteouts":0,"num_read":74491,"num_read_kb":278808,"num_write":74884,"num_write_kb":299536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"72'165","reported_seq":1850,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-03T17:48:01.504284+0000","last_change":"2023-10-03T17:40:17.641493+0000","last_active":"2023-10-03T17:48:01.504284+0000","last_peered":"2023-10-03T17:48:01.504284+0000","last_clean":"2023-10-03T17:48:01.504284+0000","last_became_active":"2023-10-03T17:34:42.278196+0000","last_became_peered":"2023-10-03T17:34:42.278196+0000","last_unstale":"2023-10-03T17:48:01.504284+0000","last_undegraded":"2023-10-03T17:48:01.504284+0000","last_fullsized":"2023-10-03T17:48:01.504284+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'30","last_scrub_stamp":"2023-10-03T17:40:17.640969+0000","last_deep_scrub":"46'30","last_deep_scrub_stamp":"2023-10-03T17:40:17.640969+0000","last_clean_scrub_stamp":"2023-10-03T17:40:17.640969+0000","objects_scrubbed":6,"log_size":165,"log_dups_size":0,"ondisk_log_size":165,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:04:02.954896+0000","scrub_duration":0.071054565,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":470,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":4758,"num_read_kb":3927,"num_write":158,"num_write_kb":133,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":903,"num_omap_keys":30,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"72'94403","reported_seq":160874,"reported_epoch":72,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T17:48:01.442705+0000","last_change":"2023-10-03T17:47:59.378970+0000","last_active":"2023-10-03T17:48:01.442705+0000","last_peered":"2023-10-03T17:48:01.442705+0000","last_clean":"2023-10-03T17:48:01.442705+0000","last_became_active":"2023-10-03T17:34:48.310912+0000","last_became_peered":"2023-10-03T17:34:48.310912+0000","last_unstale":"2023-10-03T17:48:01.442705+0000","last_undegraded":"2023-10-03T17:48:01.442705+0000","last_fullsized":"2023-10-03T17:48:01.442705+0000","mapping_epoch":16,"log_start":"66'84397","ondisk_log_start":"66'84397","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'58196","last_scrub_stamp":"2023-10-03T17:43:26.193075+0000","last_deep_scrub":"53'58196","last_deep_scrub_stamp":"2023-10-03T17:43:26.193075+0000","last_clean_scrub_stamp":"2023-10-03T17:43:26.193075+0000","objects_scrubbed":2326,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":11,"last_scrub_duration":196,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T19:51:04.235427+0000","scrub_duration":195.44739730800001,"objects_trimmed":2,"snaptrim_duration":55.976642611000003,"stat_sum":{"num_bytes":5398872064,"num_objects":6309,"num_object_clones":5038,"num_object_copies":12618,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6309,"num_whiteouts":0,"num_read":76509,"num_read_kb":287960,"num_write":77143,"num_write_kb":308572,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"54","length":"1e"},{"start":"7c","length":"8"},{"start":"8a","length":"2"},{"start":"90","length":"a"},{"start":"a4","length":"9"}]},{"pgid":"2.0","version":"71'88","reported_seq":739,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-03T17:48:00.510170+0000","last_change":"2023-10-03T17:40:10.708344+0000","last_active":"2023-10-03T17:48:00.510170+0000","last_peered":"2023-10-03T17:48:00.510170+0000","last_clean":"2023-10-03T17:48:00.510170+0000","last_became_active":"2023-10-03T17:34:42.272306+0000","last_became_peered":"2023-10-03T17:34:42.272306+0000","last_unstale":"2023-10-03T17:48:00.510170+0000","last_undegraded":"2023-10-03T17:48:00.510170+0000","last_fullsized":"2023-10-03T17:48:00.510170+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'20","last_scrub_stamp":"2023-10-03T17:40:10.708060+0000","last_deep_scrub":"46'20","last_deep_scrub_stamp":"2023-10-03T17:40:10.708060+0000","last_clean_scrub_stamp":"2023-10-03T17:40:10.708060+0000","objects_scrubbed":7,"log_size":88,"log_dups_size":0,"ondisk_log_size":88,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:03:09.097252+0000","scrub_duration":0.042279491000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":755,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":1674,"num_read_kb":1364,"num_write":82,"num_write_kb":68,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":301,"num_omap_keys":10,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"72'94851","reported_seq":161933,"reported_epoch":72,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T17:48:01.698008+0000","last_change":"2023-10-03T17:48:01.619059+0000","last_active":"2023-10-03T17:48:01.698008+0000","last_peered":"2023-10-03T17:48:01.698008+0000","last_clean":"2023-10-03T17:48:01.698008+0000","last_became_active":"2023-10-03T17:34:48.310612+0000","last_became_peered":"2023-10-03T17:34:48.310612+0000","last_unstale":"2023-10-03T17:48:01.698008+0000","last_undegraded":"2023-10-03T17:48:01.698008+0000","last_fullsized":"2023-10-03T17:48:01.698008+0000","mapping_epoch":16,"log_start":"66'84769","ondisk_log_start":"66'84769","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'75807","last_scrub_stamp":"2023-10-03T17:45:43.833310+0000","last_deep_scrub":"61'75807","last_deep_scrub_stamp":"2023-10-03T17:45:43.833310+0000","last_clean_scrub_stamp":"2023-10-03T17:45:43.833310+0000","objects_scrubbed":3607,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":25,"last_scrub_duration":315,"scrub_schedule":"queued for deep scrub","scrub_duration":314.26374841099999,"objects_trimmed":7,"snaptrim_duration":0.26932080200000003,"stat_sum":{"num_bytes":5466538003,"num_objects":7376,"num_object_clones":6088,"num_object_copies":14752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7376,"num_whiteouts":0,"num_read":76776,"num_read_kb":288388,"num_write":78073,"num_write_kb":312286,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"54","length":"1e"},{"start":"7c","length":"8"},{"start":"8a","length":"2"},{"start":"90","length":"5"}]},{"pgid":"2.3","version":"69'38","reported_seq":147,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-03T17:48:00.340989+0000","last_change":"2023-10-03T17:40:24.472734+0000","last_active":"2023-10-03T17:48:00.340989+0000","last_peered":"2023-10-03T17:48:00.340989+0000","last_clean":"2023-10-03T17:48:00.340989+0000","last_became_active":"2023-10-03T17:34:42.270194+0000","last_became_peered":"2023-10-03T17:34:42.270194+0000","last_unstale":"2023-10-03T17:48:00.340989+0000","last_undegraded":"2023-10-03T17:48:00.340989+0000","last_fullsized":"2023-10-03T17:48:00.340989+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'8","last_scrub_stamp":"2023-10-03T17:40:24.472573+0000","last_deep_scrub":"46'8","last_deep_scrub_stamp":"2023-10-03T17:40:24.472573+0000","last_clean_scrub_stamp":"2023-10-03T17:40:24.472573+0000","objects_scrubbed":4,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T18:28:04.735033+0000","scrub_duration":0.024990637999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":38,"num_read_kb":20,"num_write":34,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"46'99","reported_seq":215,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-03T17:48:00.341370+0000","last_change":"2023-10-03T17:40:10.452341+0000","last_active":"2023-10-03T17:48:00.341370+0000","last_peered":"2023-10-03T17:48:00.341370+0000","last_clean":"2023-10-03T17:48:00.341370+0000","last_became_active":"2023-10-03T17:34:39.502393+0000","last_became_peered":"2023-10-03T17:34:39.502393+0000","last_unstale":"2023-10-03T17:48:00.341370+0000","last_undegraded":"2023-10-03T17:48:00.341370+0000","last_fullsized":"2023-10-03T17:48:00.341370+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'99","last_scrub_stamp":"2023-10-03T17:40:10.449099+0000","last_deep_scrub":"46'99","last_deep_scrub_stamp":"2023-10-03T17:40:10.449099+0000","last_clean_scrub_stamp":"2023-10-03T17:40:10.449099+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T20:18:04.958666+0000","scrub_duration":0.043225012,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":21744857107,"num_objects":31808,"num_object_clones":26687,"num_object_copies":63616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31808,"num_whiteouts":0,"num_read":306856,"num_read_kb":1151868,"num_write":310346,"num_write_kb":1241378,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2542387200,"data_stored":12000047142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40137,"ondisk_log_size":40137,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":5372,"num_objects":143,"num_object_clones":0,"num_object_copies":286,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":143,"num_whiteouts":0,"num_read":34406,"num_read_kb":28717,"num_write":1290,"num_write_kb":1082,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":983040,"data_stored":10744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9073,"ondisk_log_size":9073,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":7460,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738535,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3094460,"kb_used_data":1241584,"kb_used_omap":1,"kb_used_meta":1852862,"kb_avail":91277380,"statfs":{"total":96636764160,"available":93468037120,"internally_reserved":0,"allocated":1271382016,"data_stored":5991181177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1897331142},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,0,3,6,10,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 17:48:01 2023","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.42199999999999999,"15min":0.45200000000000001},"min":{"1min":0.17199999999999999,"5min":0.16500000000000001,"15min":0.14299999999999999},"max":{"1min":3.9950000000000001,"5min":3.9950000000000001,"15min":4.4640000000000004},"last":0.188},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.379,"15min":0.442},"min":{"1min":0.186,"5min":0.17799999999999999,"15min":0.17100000000000001},"max":{"1min":2.9049999999999998,"5min":2.9049999999999998,"15min":4.4180000000000001},"last":0.29699999999999999}]},{"osd":1,"last update":"Tue Oct 3 17:48:01 2023","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.54700000000000004,"15min":0.51900000000000002},"min":{"1min":0.222,"5min":0.184,"15min":0.16600000000000001},"max":{"1min":4.0359999999999996,"5min":8.1059999999999999,"15min":8.1059999999999999},"last":0.222},{"interface":"front","average":{"1min":0.42599999999999999,"5min":0.46700000000000003,"15min":0.47199999999999998},"min":{"1min":0.23599999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.92400000000000004,"5min":4.1120000000000001,"15min":4.1120000000000001},"last":0.39400000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738533,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1557116,"kb_used_data":612528,"kb_used_omap":1,"kb_used_meta":944574,"kb_avail":92814724,"statfs":{"total":96636764160,"available":95042277376,"internally_reserved":0,"allocated":627228672,"data_stored":3251416088,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":967244230},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,2,6,8,9,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 17:47:09 2023","interfaces":[{"interface":"back","average":{"1min":0.26700000000000002,"5min":0.30599999999999999,"15min":0.38400000000000001},"min":{"1min":0.20599999999999999,"5min":0.17199999999999999,"15min":0.161},"max":{"1min":0.46400000000000002,"5min":2.3809999999999998,"15min":4.1879999999999997},"last":0.28000000000000003},{"interface":"front","average":{"1min":0.28999999999999998,"5min":0.43099999999999999,"15min":0.42699999999999999},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.52900000000000003,"5min":4.5759999999999996,"15min":4.5759999999999996},"last":5.8959999999999999}]},{"osd":2,"last update":"Tue Oct 3 17:47:09 2023","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.41199999999999998,"15min":0.42899999999999999},"min":{"1min":0.19800000000000001,"5min":0.187,"15min":0.159},"max":{"1min":2.613,"5min":2.613,"15min":3.0390000000000001},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.376,"15min":0.42499999999999999},"min":{"1min":0.23499999999999999,"5min":0.16400000000000001,"15min":0.16400000000000001},"max":{"1min":2.6970000000000001,"5min":2.6970000000000001,"15min":3.0699999999999998},"last":0.38300000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738533,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1532984,"kb_used_data":629996,"kb_used_omap":1,"kb_used_meta":902974,"kb_avail":92838856,"statfs":{"total":96636764160,"available":95066988544,"internally_reserved":0,"allocated":645115904,"data_stored":2733540894,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":924645830},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,4,11,2,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 17:48:00 2023","interfaces":[{"interface":"back","average":{"1min":0.318,"5min":0.373,"15min":0.41899999999999998},"min":{"1min":0.185,"5min":0.17100000000000001,"15min":0.154},"max":{"1min":0.83399999999999996,"5min":3.4649999999999999,"15min":5.7649999999999997},"last":0.23999999999999999},{"interface":"front","average":{"1min":0.34699999999999998,"5min":0.38300000000000001,"15min":0.46200000000000002},"min":{"1min":0.20999999999999999,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":0.92600000000000005,"5min":3.4279999999999999,"15min":9.4770000000000003},"last":0.28100000000000003}]},{"osd":2,"last update":"Tue Oct 3 17:48:00 2023","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.53200000000000003,"15min":0.52600000000000002},"min":{"1min":0.216,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.95799999999999996,"5min":4.1619999999999999,"15min":9.4149999999999991},"last":0.309},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.61099999999999999,"15min":0.50900000000000001},"min":{"1min":0.246,"5min":0.19700000000000001,"15min":0.189},"max":{"1min":0.89900000000000002,"5min":5.218,"15min":5.218},"last":0.35399999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":339968,"data_stored":3711,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":5372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":1661,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":644681728,"data_stored":2740502528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":626511872,"data_stored":3260051475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1271193600,"data_stored":5999493139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-03T17:48:07.956 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-03T17:34:47.281095+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=34, tm_sec=47, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=40, tm_sec=6, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T17:48:07.956 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T17:48:22.796 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:48:22.797+0000 7f09e4851640 -1 reset not still connected to 0x557d76253d40 2023-10-03T17:48:22.796 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:48:22.801+0000 7f09e4851640 -1 reset not still connected to 0x557d76621040 2023-10-03T17:48:22.797 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:48:22.801+0000 7f09e4851640 -1 reset not still connected to 0x557d7664e750 2023-10-03T17:48:22.797 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:48:22.801+0000 7f09e4851640 -1 reset not still connected to 0x557d7664ea90 2023-10-03T17:48:22.799 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:48:22.801+0000 7f09e4851640 -1 reset not still connected to 0x557d7664f380 2023-10-03T17:48:22.799 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:48:22.801+0000 7f09e4851640 -1 reset not still connected to 0x557d76681ba0 2023-10-03T17:48:22.799 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:48:22.801+0000 7f09e4851640 -1 reset not still connected to 0x557d766b56c0 2023-10-03T17:48:22.800 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:48:22.801+0000 7f09e4851640 -1 reset not still connected to 0x557d76eca270 2023-10-03T17:48:22.800 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:48:22.801+0000 7f09e4851640 -1 reset not still connected to 0x557d76f93ba0 2023-10-03T17:48:27.957 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T17:48:28.508 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-03T17:48:28.509 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-10-03T17:48:28.543 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":488,"stamp":"2023-10-03T17:48:28.140315+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":21779883823,"num_objects":34391,"num_object_clones":29125,"num_object_copies":68782,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34391,"num_whiteouts":0,"num_read":350734,"num_read_kb":1209946,"num_write":318636,"num_write_kb":1271435,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":49658,"ondisk_log_size":49658,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":6474024,"kb_used_data":2538508,"kb_used_omap":2894,"kb_used_meta":3932593,"kb_avail":276641496,"statfs":{"total":289910292480,"available":283280891904,"internally_reserved":0,"allocated":2599432192,"data_stored":13332626890,"data_compressed":7460,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":2964133,"internal_metadata":4026975579},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,10,5,10,11,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":21360640,"num_objects":940,"num_object_clones":940,"num_object_copies":1880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":940,"num_whiteouts":0,"num_read":5581,"num_read_kb":17475,"num_write":4107,"num_write_kb":16428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002006"},"pg_stats":[{"pgid":"2.7","version":"74'1684","reported_seq":3476,"reported_epoch":74,"state":"active+clean","last_fresh":"2023-10-03T17:48:26.041871+0000","last_change":"2023-10-03T17:40:17.407011+0000","last_active":"2023-10-03T17:48:26.041871+0000","last_peered":"2023-10-03T17:48:26.041871+0000","last_clean":"2023-10-03T17:48:26.041871+0000","last_became_active":"2023-10-03T17:34:42.276624+0000","last_became_peered":"2023-10-03T17:34:42.276624+0000","last_unstale":"2023-10-03T17:48:26.041871+0000","last_undegraded":"2023-10-03T17:48:26.041871+0000","last_fullsized":"2023-10-03T17:48:26.041871+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'673","last_scrub_stamp":"2023-10-03T17:40:17.406813+0000","last_deep_scrub":"46'673","last_deep_scrub_stamp":"2023-10-03T17:40:17.406813+0000","last_clean_scrub_stamp":"2023-10-03T17:40:17.406813+0000","objects_scrubbed":7,"log_size":1684,"log_dups_size":0,"ondisk_log_size":1684,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T21:38:35.056889+0000","scrub_duration":0.013778076,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":737,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":5145,"num_read_kb":4245,"num_write":196,"num_write_kb":160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":903,"num_omap_keys":30,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"74'2807","reported_seq":6967,"reported_epoch":74,"state":"active+clean","last_fresh":"2023-10-03T17:48:26.045548+0000","last_change":"2023-10-03T17:40:13.376519+0000","last_active":"2023-10-03T17:48:26.045548+0000","last_peered":"2023-10-03T17:48:26.045548+0000","last_clean":"2023-10-03T17:48:26.045548+0000","last_became_active":"2023-10-03T17:34:42.271201+0000","last_became_peered":"2023-10-03T17:34:42.271201+0000","last_unstale":"2023-10-03T17:48:26.045548+0000","last_undegraded":"2023-10-03T17:48:26.045548+0000","last_fullsized":"2023-10-03T17:48:26.045548+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'1129","last_scrub_stamp":"2023-10-03T17:40:13.376332+0000","last_deep_scrub":"46'1129","last_deep_scrub_stamp":"2023-10-03T17:40:13.376332+0000","last_clean_scrub_stamp":"2023-10-03T17:40:13.376332+0000","objects_scrubbed":11,"log_size":2807,"log_dups_size":0,"ondisk_log_size":2807,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T18:59:17.088585+0000","scrub_duration":0.015998675,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":673,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":10306,"num_read_kb":8815,"num_write":347,"num_write_kb":292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2912,"num_omap_keys":93,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"73'41","reported_seq":152,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-03T17:48:02.548268+0000","last_change":"2023-10-03T17:40:25.496010+0000","last_active":"2023-10-03T17:48:02.548268+0000","last_peered":"2023-10-03T17:48:02.548268+0000","last_clean":"2023-10-03T17:48:02.548268+0000","last_became_active":"2023-10-03T17:34:42.276941+0000","last_became_peered":"2023-10-03T17:34:42.276941+0000","last_unstale":"2023-10-03T17:48:02.548268+0000","last_undegraded":"2023-10-03T17:48:02.548268+0000","last_fullsized":"2023-10-03T17:48:02.548268+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'14","last_scrub_stamp":"2023-10-03T17:40:25.495827+0000","last_deep_scrub":"46'14","last_deep_scrub_stamp":"2023-10-03T17:40:25.495827+0000","last_clean_scrub_stamp":"2023-10-03T17:40:25.495827+0000","objects_scrubbed":7,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:05:44.939663+0000","scrub_duration":0.0133078,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1109,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":44,"num_read_kb":17,"num_write":34,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"74'2237","reported_seq":5238,"reported_epoch":74,"state":"active+clean","last_fresh":"2023-10-03T17:48:26.038872+0000","last_change":"2023-10-03T17:40:26.513902+0000","last_active":"2023-10-03T17:48:26.038872+0000","last_peered":"2023-10-03T17:48:26.038872+0000","last_clean":"2023-10-03T17:48:26.038872+0000","last_became_active":"2023-10-03T17:34:42.276459+0000","last_became_peered":"2023-10-03T17:34:42.276459+0000","last_unstale":"2023-10-03T17:48:26.038872+0000","last_undegraded":"2023-10-03T17:48:26.038872+0000","last_fullsized":"2023-10-03T17:48:26.038872+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'964","last_scrub_stamp":"2023-10-03T17:40:26.513729+0000","last_deep_scrub":"46'964","last_deep_scrub_stamp":"2023-10-03T17:40:26.513729+0000","last_clean_scrub_stamp":"2023-10-03T17:40:26.513729+0000","objects_scrubbed":10,"log_size":2237,"log_dups_size":0,"ondisk_log_size":2237,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:41:33.182751+0000","scrub_duration":0.018190280999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":7528,"num_read_kb":6295,"num_write":248,"num_write_kb":210,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1731,"num_omap_keys":80,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"74'99067","reported_seq":169135,"reported_epoch":74,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T17:48:26.630678+0000","last_change":"2023-10-03T17:48:25.102096+0000","last_active":"2023-10-03T17:48:26.630678+0000","last_peered":"2023-10-03T17:48:26.630678+0000","last_clean":"2023-10-03T17:48:26.630678+0000","last_became_active":"2023-10-03T17:34:48.310701+0000","last_became_peered":"2023-10-03T17:34:48.310701+0000","last_unstale":"2023-10-03T17:48:26.630678+0000","last_undegraded":"2023-10-03T17:48:26.630678+0000","last_fullsized":"2023-10-03T17:48:26.630678+0000","mapping_epoch":16,"log_start":"70'89065","ondisk_log_start":"70'89065","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'80695","last_scrub_stamp":"2023-10-03T17:46:13.982571+0000","last_deep_scrub":"66'80695","last_deep_scrub_stamp":"2023-10-03T17:46:13.982571+0000","last_clean_scrub_stamp":"2023-10-03T17:46:13.982571+0000","objects_scrubbed":3410,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":42,"last_scrub_duration":363,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T19:02:07.971209+0000","scrub_duration":362.26574615599998,"objects_trimmed":318,"snaptrim_duration":21.727629868000001,"stat_sum":{"num_bytes":5613408256,"num_objects":8902,"num_object_clones":7582,"num_object_copies":17804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8902,"num_whiteouts":0,"num_read":80780,"num_read_kb":303384,"num_write":81999,"num_write_kb":327996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"54","length":"1e"},{"start":"7c","length":"8"}]},{"pgid":"2.2","version":"74'2307","reported_seq":4704,"reported_epoch":74,"state":"active+clean","last_fresh":"2023-10-03T17:48:26.491668+0000","last_change":"2023-10-03T17:40:08.675119+0000","last_active":"2023-10-03T17:48:26.491668+0000","last_peered":"2023-10-03T17:48:26.491668+0000","last_clean":"2023-10-03T17:48:26.491668+0000","last_became_active":"2023-10-03T17:34:42.270625+0000","last_became_peered":"2023-10-03T17:34:42.270625+0000","last_unstale":"2023-10-03T17:48:26.491668+0000","last_undegraded":"2023-10-03T17:48:26.491668+0000","last_fullsized":"2023-10-03T17:48:26.491668+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'841","last_scrub_stamp":"2023-10-03T17:40:08.674910+0000","last_deep_scrub":"46'841","last_deep_scrub_stamp":"2023-10-03T17:40:08.674910+0000","last_clean_scrub_stamp":"2023-10-03T17:40:08.674910+0000","objects_scrubbed":11,"log_size":2307,"log_dups_size":0,"ondisk_log_size":2307,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T21:05:42.790614+0000","scrub_duration":0.039189555000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":908,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":6889,"num_read_kb":5698,"num_write":274,"num_write_kb":229,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2564,"num_omap_keys":60,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"74'89979","reported_seq":157156,"reported_epoch":74,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T17:48:26.900320+0000","last_change":"2023-10-03T17:43:28.494717+0000","last_active":"2023-10-03T17:48:26.900320+0000","last_peered":"2023-10-03T17:48:26.900320+0000","last_clean":"2023-10-03T17:48:26.900320+0000","last_became_active":"2023-10-03T17:34:48.310555+0000","last_became_peered":"2023-10-03T17:34:48.310555+0000","last_unstale":"2023-10-03T17:48:26.900320+0000","last_undegraded":"2023-10-03T17:48:26.900320+0000","last_fullsized":"2023-10-03T17:48:26.900320+0000","mapping_epoch":16,"log_start":"66'79892","ondisk_log_start":"66'79892","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_clean_scrub_stamp":"2023-10-03T17:34:47.281095+0000","objects_scrubbed":4735,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":80,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 298s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086289699999999995,"stat_sum":{"num_bytes":5282938880,"num_objects":10717,"num_object_clones":9475,"num_object_copies":21434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10717,"num_whiteouts":0,"num_read":76211,"num_read_kb":285544,"num_write":76539,"num_write_kb":306156,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"74'178","reported_seq":1960,"reported_epoch":74,"state":"active+clean","last_fresh":"2023-10-03T17:48:25.526198+0000","last_change":"2023-10-03T17:40:17.641493+0000","last_active":"2023-10-03T17:48:25.526198+0000","last_peered":"2023-10-03T17:48:25.526198+0000","last_clean":"2023-10-03T17:48:25.526198+0000","last_became_active":"2023-10-03T17:34:42.278196+0000","last_became_peered":"2023-10-03T17:34:42.278196+0000","last_unstale":"2023-10-03T17:48:25.526198+0000","last_undegraded":"2023-10-03T17:48:25.526198+0000","last_fullsized":"2023-10-03T17:48:25.526198+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'30","last_scrub_stamp":"2023-10-03T17:40:17.640969+0000","last_deep_scrub":"46'30","last_deep_scrub_stamp":"2023-10-03T17:40:17.640969+0000","last_clean_scrub_stamp":"2023-10-03T17:40:17.640969+0000","objects_scrubbed":6,"log_size":178,"log_dups_size":0,"ondisk_log_size":178,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:04:02.954896+0000","scrub_duration":0.071054565,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":470,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":5125,"num_read_kb":4234,"num_write":178,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":903,"num_omap_keys":30,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"74'97820","reported_seq":164961,"reported_epoch":74,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T17:48:26.652815+0000","last_change":"2023-10-03T17:48:23.880695+0000","last_active":"2023-10-03T17:48:26.652815+0000","last_peered":"2023-10-03T17:48:26.652815+0000","last_clean":"2023-10-03T17:48:26.652815+0000","last_became_active":"2023-10-03T17:34:48.310912+0000","last_became_peered":"2023-10-03T17:34:48.310912+0000","last_unstale":"2023-10-03T17:48:26.652815+0000","last_undegraded":"2023-10-03T17:48:26.652815+0000","last_fullsized":"2023-10-03T17:48:26.652815+0000","mapping_epoch":16,"log_start":"70'87819","ondisk_log_start":"70'87819","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'58196","last_scrub_stamp":"2023-10-03T17:43:26.193075+0000","last_deep_scrub":"53'58196","last_deep_scrub_stamp":"2023-10-03T17:43:26.193075+0000","last_clean_scrub_stamp":"2023-10-03T17:43:26.193075+0000","objects_scrubbed":2326,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":13,"last_scrub_duration":196,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T21:42:26.771584+0000","scrub_duration":195.44739730800001,"objects_trimmed":432,"snaptrim_duration":20.511664,"stat_sum":{"num_bytes":5407313920,"num_objects":6710,"num_object_clones":5439,"num_object_copies":13420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6710,"num_whiteouts":0,"num_read":78252,"num_read_kb":294820,"num_write":78747,"num_write_kb":314988,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"54","length":"1e"},{"start":"7c","length":"8"},{"start":"8a","length":"2"},{"start":"90","length":"a"},{"start":"a4","length":"a"},{"start":"b8","length":"7"}]},{"pgid":"2.0","version":"73'95","reported_seq":772,"reported_epoch":74,"state":"active+clean","last_fresh":"2023-10-03T17:48:20.407831+0000","last_change":"2023-10-03T17:40:10.708344+0000","last_active":"2023-10-03T17:48:20.407831+0000","last_peered":"2023-10-03T17:48:20.407831+0000","last_clean":"2023-10-03T17:48:20.407831+0000","last_became_active":"2023-10-03T17:34:42.272306+0000","last_became_peered":"2023-10-03T17:34:42.272306+0000","last_unstale":"2023-10-03T17:48:20.407831+0000","last_undegraded":"2023-10-03T17:48:20.407831+0000","last_fullsized":"2023-10-03T17:48:20.407831+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'20","last_scrub_stamp":"2023-10-03T17:40:10.708060+0000","last_deep_scrub":"46'20","last_deep_scrub_stamp":"2023-10-03T17:40:10.708060+0000","last_clean_scrub_stamp":"2023-10-03T17:40:10.708060+0000","objects_scrubbed":7,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:03:09.097252+0000","scrub_duration":0.042279491000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":806,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":1751,"num_read_kb":1430,"num_write":90,"num_write_kb":72,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":301,"num_omap_keys":10,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"74'98276","reported_seq":166049,"reported_epoch":74,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T17:48:26.909631+0000","last_change":"2023-10-03T17:48:23.122130+0000","last_active":"2023-10-03T17:48:26.909631+0000","last_peered":"2023-10-03T17:48:26.909631+0000","last_clean":"2023-10-03T17:48:26.909631+0000","last_became_active":"2023-10-03T17:34:48.310612+0000","last_became_peered":"2023-10-03T17:34:48.310612+0000","last_unstale":"2023-10-03T17:48:26.909631+0000","last_undegraded":"2023-10-03T17:48:26.909631+0000","last_fullsized":"2023-10-03T17:48:26.909631+0000","mapping_epoch":16,"log_start":"70'88196","ondisk_log_start":"70'88196","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'75807","last_scrub_stamp":"2023-10-03T17:45:43.833310+0000","last_deep_scrub":"61'75807","last_deep_scrub_stamp":"2023-10-03T17:45:43.833310+0000","last_clean_scrub_stamp":"2023-10-03T17:45:43.833310+0000","objects_scrubbed":3607,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":30,"last_scrub_duration":315,"scrub_schedule":"queued for deep scrub","scrub_duration":314.26374841099999,"objects_trimmed":364,"snaptrim_duration":19.762453795999999,"stat_sum":{"num_bytes":5475627027,"num_objects":7917,"num_object_clones":6629,"num_object_copies":15834,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7917,"num_whiteouts":0,"num_read":78558,"num_read_kb":295352,"num_write":79811,"num_write_kb":319238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"54","length":"1e"},{"start":"7c","length":"8"},{"start":"8a","length":"2"},{"start":"90","length":"a"}]},{"pgid":"2.3","version":"73'40","reported_seq":152,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-03T17:48:03.064718+0000","last_change":"2023-10-03T17:40:24.472734+0000","last_active":"2023-10-03T17:48:03.064718+0000","last_peered":"2023-10-03T17:48:03.064718+0000","last_clean":"2023-10-03T17:48:03.064718+0000","last_became_active":"2023-10-03T17:34:42.270194+0000","last_became_peered":"2023-10-03T17:34:42.270194+0000","last_unstale":"2023-10-03T17:48:03.064718+0000","last_undegraded":"2023-10-03T17:48:03.064718+0000","last_fullsized":"2023-10-03T17:48:03.064718+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'8","last_scrub_stamp":"2023-10-03T17:40:24.472573+0000","last_deep_scrub":"46'8","last_deep_scrub_stamp":"2023-10-03T17:40:24.472573+0000","last_clean_scrub_stamp":"2023-10-03T17:40:24.472573+0000","objects_scrubbed":4,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T18:28:04.735033+0000","scrub_duration":0.024990637999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":39,"num_read_kb":21,"num_write":36,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"46'99","reported_seq":217,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-03T17:48:02.354692+0000","last_change":"2023-10-03T17:40:10.452341+0000","last_active":"2023-10-03T17:48:02.354692+0000","last_peered":"2023-10-03T17:48:02.354692+0000","last_clean":"2023-10-03T17:48:02.354692+0000","last_became_active":"2023-10-03T17:34:39.502393+0000","last_became_peered":"2023-10-03T17:34:39.502393+0000","last_unstale":"2023-10-03T17:48:02.354692+0000","last_undegraded":"2023-10-03T17:48:02.354692+0000","last_fullsized":"2023-10-03T17:48:02.354692+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'99","last_scrub_stamp":"2023-10-03T17:40:10.449099+0000","last_deep_scrub":"46'99","last_deep_scrub_stamp":"2023-10-03T17:40:10.449099+0000","last_clean_scrub_stamp":"2023-10-03T17:40:10.449099+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T20:18:04.958666+0000","scrub_duration":0.043225012,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":21779288083,"num_objects":34246,"num_object_clones":29125,"num_object_copies":68492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34246,"num_whiteouts":0,"num_read":313801,"num_read_kb":1179100,"num_write":317096,"num_write_kb":1268378,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2597822464,"data_stored":13342187558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40170,"ondisk_log_size":40170,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":5372,"num_objects":143,"num_object_clones":0,"num_object_copies":286,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":143,"num_whiteouts":0,"num_read":36827,"num_read_kb":30755,"num_write":1403,"num_write_kb":1163,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":983040,"data_stored":10744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7029,"internal_metadata":0},"log_size":9389,"ondisk_log_size":9389,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":7460,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738540,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3230092,"kb_used_data":1268420,"kb_used_omap":2891,"kb_used_meta":1958772,"kb_avail":91141748,"statfs":{"total":96636764160,"available":93329149952,"internally_reserved":0,"allocated":1298862080,"data_stored":6665022439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2960945,"internal_metadata":2005782991},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,1,4,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 17:48:01 2023","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.42199999999999999,"15min":0.45200000000000001},"min":{"1min":0.17199999999999999,"5min":0.16500000000000001,"15min":0.14299999999999999},"max":{"1min":3.9950000000000001,"5min":3.9950000000000001,"15min":4.4640000000000004},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.379,"15min":0.442},"min":{"1min":0.186,"5min":0.17799999999999999,"15min":0.17100000000000001},"max":{"1min":2.9049999999999998,"5min":2.9049999999999998,"15min":4.4180000000000001},"last":0.41499999999999998}]},{"osd":1,"last update":"Tue Oct 3 17:48:01 2023","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.54700000000000004,"15min":0.51900000000000002},"min":{"1min":0.222,"5min":0.184,"15min":0.16600000000000001},"max":{"1min":4.0359999999999996,"5min":8.1059999999999999,"15min":8.1059999999999999},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.42599999999999999,"5min":0.46700000000000003,"15min":0.47199999999999998},"min":{"1min":0.23599999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.92400000000000004,"5min":4.1120000000000001,"15min":4.1120000000000001},"last":0.45600000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738538,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1624880,"kb_used_data":626216,"kb_used_omap":1,"kb_used_meta":998654,"kb_avail":92746960,"statfs":{"total":96636764160,"available":94972887040,"internally_reserved":0,"allocated":641245184,"data_stored":3636456531,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":1022622150},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,6,0,4,5,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 17:48:14 2023","interfaces":[{"interface":"back","average":{"1min":0.32500000000000001,"5min":0.32600000000000001,"15min":0.371},"min":{"1min":0.17999999999999999,"5min":0.17199999999999999,"15min":0.161},"max":{"1min":1.895,"5min":2.3809999999999998,"15min":4.1879999999999997},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.47699999999999998,"15min":0.42999999999999999},"min":{"1min":0.191,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":5.8959999999999999,"5min":5.8959999999999999,"15min":5.8959999999999999},"last":0.224}]},{"osd":2,"last update":"Tue Oct 3 17:48:14 2023","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.441,"15min":0.42299999999999999},"min":{"1min":0.184,"5min":0.184,"15min":0.159},"max":{"1min":2.6240000000000001,"5min":2.6240000000000001,"15min":3.0390000000000001},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.39700000000000002,"15min":0.41199999999999998},"min":{"1min":0.23200000000000001,"5min":0.16400000000000001,"15min":0.16400000000000001},"max":{"1min":1.9330000000000001,"5min":2.6970000000000001,"15min":3.0699999999999998},"last":0.33400000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738538,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1619052,"kb_used_data":643872,"kb_used_omap":1,"kb_used_meta":975166,"kb_avail":92752788,"statfs":{"total":96636764160,"available":94978854912,"internally_reserved":0,"allocated":659324928,"data_stored":3031147920,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":998570438},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,1,2,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 17:48:00 2023","interfaces":[{"interface":"back","average":{"1min":0.318,"5min":0.373,"15min":0.41899999999999998},"min":{"1min":0.185,"5min":0.17100000000000001,"15min":0.154},"max":{"1min":0.83399999999999996,"5min":3.4649999999999999,"15min":5.7649999999999997},"last":0.317},{"interface":"front","average":{"1min":0.34699999999999998,"5min":0.38300000000000001,"15min":0.46200000000000002},"min":{"1min":0.20999999999999999,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":0.92600000000000005,"5min":3.4279999999999999,"15min":9.4770000000000003},"last":1.28}]},{"osd":2,"last update":"Tue Oct 3 17:48:00 2023","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.53200000000000003,"15min":0.52600000000000002},"min":{"1min":0.216,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.95799999999999996,"5min":4.1619999999999999,"15min":9.4149999999999991},"last":1.244},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.61099999999999999,"15min":0.50900000000000001},"min":{"1min":0.246,"5min":0.19700000000000001,"15min":0.189},"max":{"1min":0.89900000000000002,"5min":5.218,"15min":5.218},"last":0.29899999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":335872,"data_stored":3660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":5372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":155648,"data_stored":1712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7029,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":658436096,"data_stored":3030458368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":640413696,"data_stored":3640356883,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1298972672,"data_stored":6671372307,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-03T17:48:28.545 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-03T17:34:47.281095+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=34, tm_sec=47, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=40, tm_sec=6, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T17:48:28.545 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.0 2023-10-03T17:48:29.199 INFO:teuthology.orchestra.run.smithi111.stderr:instructing pg 3.0 on osd.1 to deep-scrub 2023-10-03T17:48:29.234 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T17:48:30.001 INFO:tasks.ceph.mgr.x.smithi111.stderr:2023-10-03T17:48:30.005+0000 7fe48bb97640 -1 calc_pg_upmaps abort due to max <= 0 2023-10-03T17:48:37.012 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:48:37.013+0000 7f09e4851640 -1 reset not still connected to 0x557d75f2b2b0 2023-10-03T17:48:37.013 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:48:37.013+0000 7f09e4851640 -1 reset not still connected to 0x557d76253d40 2023-10-03T17:48:37.013 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:48:37.013+0000 7f09e4851640 -1 reset not still connected to 0x557d76621040 2023-10-03T17:48:37.013 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:48:37.013+0000 7f09e4851640 -1 reset not still connected to 0x557d7664e750 2023-10-03T17:48:37.013 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:48:37.013+0000 7f09e4851640 -1 reset not still connected to 0x557d7664ea90 2023-10-03T17:48:37.013 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:48:37.013+0000 7f09e4851640 -1 reset not still connected to 0x557d7664f380 2023-10-03T17:48:37.013 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:48:37.013+0000 7f09e4851640 -1 reset not still connected to 0x557d7664f930 2023-10-03T17:48:37.013 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:48:37.013+0000 7f09e4851640 -1 reset not still connected to 0x557d76681ba0 2023-10-03T17:48:37.014 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:48:37.013+0000 7f09e4851640 -1 reset not still connected to 0x557d766a40d0 2023-10-03T17:48:37.014 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:48:37.013+0000 7f09e4851640 -1 reset not still connected to 0x557d766b56c0 2023-10-03T17:48:37.014 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:48:37.013+0000 7f09e4851640 -1 reset not still connected to 0x557d76eca270 2023-10-03T17:48:37.014 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:48:37.013+0000 7f09e4851640 -1 reset not still connected to 0x557d76f93ba0 2023-10-03T17:48:41.181 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:48:41.180+0000 7f09e4851640 -1 reset not still connected to 0x557d75f2b2b0 2023-10-03T17:48:41.181 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:48:41.180+0000 7f09e4851640 -1 reset not still connected to 0x557d76253d40 2023-10-03T17:48:41.181 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:48:41.180+0000 7f09e4851640 -1 reset not still connected to 0x557d76621040 2023-10-03T17:48:41.181 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:48:41.180+0000 7f09e4851640 -1 reset not still connected to 0x557d7664e750 2023-10-03T17:48:41.182 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:48:41.180+0000 7f09e4851640 -1 reset not still connected to 0x557d7664ea90 2023-10-03T17:48:41.182 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:48:41.180+0000 7f09e4851640 -1 reset not still connected to 0x557d7664f380 2023-10-03T17:48:41.182 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:48:41.180+0000 7f09e4851640 -1 reset not still connected to 0x557d7664f930 2023-10-03T17:48:41.182 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:48:41.180+0000 7f09e4851640 -1 reset not still connected to 0x557d76681ba0 2023-10-03T17:48:41.182 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:48:41.180+0000 7f09e4851640 -1 reset not still connected to 0x557d766b56c0 2023-10-03T17:48:41.182 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:48:41.180+0000 7f09e4851640 -1 reset not still connected to 0x557d76eca270 2023-10-03T17:48:41.182 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:48:41.180+0000 7f09e4851640 -1 reset not still connected to 0x557d76f93ba0 2023-10-03T17:48:49.236 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T17:48:49.820 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-03T17:48:49.820 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-10-03T17:48:49.888 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":498,"stamp":"2023-10-03T17:48:48.147392+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":21812033327,"num_objects":34277,"num_object_clones":29011,"num_object_copies":68554,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34277,"num_whiteouts":0,"num_read":356855,"num_read_kb":1233974,"num_write":325172,"num_write_kb":1297579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":49897,"ondisk_log_size":49897,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":5285900,"kb_used_data":2591476,"kb_used_omap":2894,"kb_used_meta":2691505,"kb_avail":277829620,"statfs":{"total":289910292480,"available":284497530880,"internally_reserved":0,"allocated":2653671424,"data_stored":13392481738,"data_compressed":7460,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":2964133,"internal_metadata":2756101467},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,2,4,14,11,14,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":25255936,"num_objects":-205,"num_object_clones":-205,"num_object_copies":-410,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-205,"num_whiteouts":0,"num_read":4661,"num_read_kb":18328,"num_write":5033,"num_write_kb":20132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005663"},"pg_stats":[{"pgid":"2.7","version":"74'1720","reported_seq":3524,"reported_epoch":74,"state":"active+clean","last_fresh":"2023-10-03T17:48:46.006731+0000","last_change":"2023-10-03T17:40:17.407011+0000","last_active":"2023-10-03T17:48:46.006731+0000","last_peered":"2023-10-03T17:48:46.006731+0000","last_clean":"2023-10-03T17:48:46.006731+0000","last_became_active":"2023-10-03T17:34:42.276624+0000","last_became_peered":"2023-10-03T17:34:42.276624+0000","last_unstale":"2023-10-03T17:48:46.006731+0000","last_undegraded":"2023-10-03T17:48:46.006731+0000","last_fullsized":"2023-10-03T17:48:46.006731+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'673","last_scrub_stamp":"2023-10-03T17:40:17.406813+0000","last_deep_scrub":"46'673","last_deep_scrub_stamp":"2023-10-03T17:40:17.406813+0000","last_clean_scrub_stamp":"2023-10-03T17:40:17.406813+0000","objects_scrubbed":7,"log_size":1720,"log_dups_size":0,"ondisk_log_size":1720,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T21:38:35.056889+0000","scrub_duration":0.013778076,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":737,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":5145,"num_read_kb":4245,"num_write":196,"num_write_kb":160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":903,"num_omap_keys":30,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"74'2867","reported_seq":7047,"reported_epoch":74,"state":"active+clean","last_fresh":"2023-10-03T17:48:46.007813+0000","last_change":"2023-10-03T17:40:13.376519+0000","last_active":"2023-10-03T17:48:46.007813+0000","last_peered":"2023-10-03T17:48:46.007813+0000","last_clean":"2023-10-03T17:48:46.007813+0000","last_became_active":"2023-10-03T17:34:42.271201+0000","last_became_peered":"2023-10-03T17:34:42.271201+0000","last_unstale":"2023-10-03T17:48:46.007813+0000","last_undegraded":"2023-10-03T17:48:46.007813+0000","last_fullsized":"2023-10-03T17:48:46.007813+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'1129","last_scrub_stamp":"2023-10-03T17:40:13.376332+0000","last_deep_scrub":"46'1129","last_deep_scrub_stamp":"2023-10-03T17:40:13.376332+0000","last_clean_scrub_stamp":"2023-10-03T17:40:13.376332+0000","objects_scrubbed":11,"log_size":2867,"log_dups_size":0,"ondisk_log_size":2867,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T18:59:17.088585+0000","scrub_duration":0.015998675,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":673,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":10306,"num_read_kb":8815,"num_write":347,"num_write_kb":292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2912,"num_omap_keys":93,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"73'41","reported_seq":152,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-03T17:48:02.548268+0000","last_change":"2023-10-03T17:40:25.496010+0000","last_active":"2023-10-03T17:48:02.548268+0000","last_peered":"2023-10-03T17:48:02.548268+0000","last_clean":"2023-10-03T17:48:02.548268+0000","last_became_active":"2023-10-03T17:34:42.276941+0000","last_became_peered":"2023-10-03T17:34:42.276941+0000","last_unstale":"2023-10-03T17:48:02.548268+0000","last_undegraded":"2023-10-03T17:48:02.548268+0000","last_fullsized":"2023-10-03T17:48:02.548268+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'14","last_scrub_stamp":"2023-10-03T17:40:25.495827+0000","last_deep_scrub":"46'14","last_deep_scrub_stamp":"2023-10-03T17:40:25.495827+0000","last_clean_scrub_stamp":"2023-10-03T17:40:25.495827+0000","objects_scrubbed":7,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:05:44.939663+0000","scrub_duration":0.0133078,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1109,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":44,"num_read_kb":17,"num_write":34,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"74'2285","reported_seq":5302,"reported_epoch":74,"state":"active+clean","last_fresh":"2023-10-03T17:48:46.007003+0000","last_change":"2023-10-03T17:40:26.513902+0000","last_active":"2023-10-03T17:48:46.007003+0000","last_peered":"2023-10-03T17:48:46.007003+0000","last_clean":"2023-10-03T17:48:46.007003+0000","last_became_active":"2023-10-03T17:34:42.276459+0000","last_became_peered":"2023-10-03T17:34:42.276459+0000","last_unstale":"2023-10-03T17:48:46.007003+0000","last_undegraded":"2023-10-03T17:48:46.007003+0000","last_fullsized":"2023-10-03T17:48:46.007003+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'964","last_scrub_stamp":"2023-10-03T17:40:26.513729+0000","last_deep_scrub":"46'964","last_deep_scrub_stamp":"2023-10-03T17:40:26.513729+0000","last_clean_scrub_stamp":"2023-10-03T17:40:26.513729+0000","objects_scrubbed":10,"log_size":2285,"log_dups_size":0,"ondisk_log_size":2285,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:41:33.182751+0000","scrub_duration":0.018190280999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":7528,"num_read_kb":6295,"num_write":248,"num_write_kb":210,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1731,"num_omap_keys":80,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"74'101732","reported_seq":173024,"reported_epoch":74,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T17:48:46.896103+0000","last_change":"2023-10-03T17:48:46.680281+0000","last_active":"2023-10-03T17:48:46.896103+0000","last_peered":"2023-10-03T17:48:46.896103+0000","last_clean":"2023-10-03T17:48:46.896103+0000","last_became_active":"2023-10-03T17:34:48.310701+0000","last_became_peered":"2023-10-03T17:34:48.310701+0000","last_unstale":"2023-10-03T17:48:46.896103+0000","last_undegraded":"2023-10-03T17:48:46.896103+0000","last_fullsized":"2023-10-03T17:48:46.896103+0000","mapping_epoch":16,"log_start":"70'91674","ondisk_log_start":"70'91674","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'80695","last_scrub_stamp":"2023-10-03T17:46:13.982571+0000","last_deep_scrub":"66'80695","last_deep_scrub_stamp":"2023-10-03T17:46:13.982571+0000","last_clean_scrub_stamp":"2023-10-03T17:46:13.982571+0000","objects_scrubbed":3410,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":36,"last_scrub_duration":363,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T19:02:07.971209+0000","scrub_duration":362.26574615599998,"objects_trimmed":691,"snaptrim_duration":43.293310626,"stat_sum":{"num_bytes":5621297152,"num_objects":8761,"num_object_clones":7441,"num_object_copies":17522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8761,"num_whiteouts":0,"num_read":82392,"num_read_kb":309720,"num_write":83686,"num_write_kb":334744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"54","length":"1e"},{"start":"7c","length":"8"},{"start":"8a","length":"2"},{"start":"90","length":"4"}]},{"pgid":"2.2","version":"74'2359","reported_seq":4772,"reported_epoch":74,"state":"active+clean","last_fresh":"2023-10-03T17:48:46.194195+0000","last_change":"2023-10-03T17:40:08.675119+0000","last_active":"2023-10-03T17:48:46.194195+0000","last_peered":"2023-10-03T17:48:46.194195+0000","last_clean":"2023-10-03T17:48:46.194195+0000","last_became_active":"2023-10-03T17:34:42.270625+0000","last_became_peered":"2023-10-03T17:34:42.270625+0000","last_unstale":"2023-10-03T17:48:46.194195+0000","last_undegraded":"2023-10-03T17:48:46.194195+0000","last_fullsized":"2023-10-03T17:48:46.194195+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'841","last_scrub_stamp":"2023-10-03T17:40:08.674910+0000","last_deep_scrub":"46'841","last_deep_scrub_stamp":"2023-10-03T17:40:08.674910+0000","last_clean_scrub_stamp":"2023-10-03T17:40:08.674910+0000","objects_scrubbed":11,"log_size":2359,"log_dups_size":0,"ondisk_log_size":2359,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T21:05:42.790614+0000","scrub_duration":0.039189555000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":908,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":6889,"num_read_kb":5698,"num_write":274,"num_write_kb":229,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2564,"num_omap_keys":60,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"74'92014","reported_seq":160434,"reported_epoch":74,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T17:48:47.054493+0000","last_change":"2023-10-03T17:43:28.494717+0000","last_active":"2023-10-03T17:48:47.054493+0000","last_peered":"2023-10-03T17:48:47.054493+0000","last_clean":"2023-10-03T17:48:47.054493+0000","last_became_active":"2023-10-03T17:34:48.310555+0000","last_became_peered":"2023-10-03T17:34:48.310555+0000","last_unstale":"2023-10-03T17:48:47.054493+0000","last_undegraded":"2023-10-03T17:48:47.054493+0000","last_fullsized":"2023-10-03T17:48:47.054493+0000","mapping_epoch":16,"log_start":"70'81993","ondisk_log_start":"70'81993","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_clean_scrub_stamp":"2023-10-03T17:34:47.281095+0000","objects_scrubbed":5006,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":80,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 318s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086289699999999995,"stat_sum":{"num_bytes":5291024384,"num_objects":10954,"num_object_clones":9712,"num_object_copies":21908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10954,"num_whiteouts":0,"num_read":77673,"num_read_kb":291268,"num_write":78093,"num_write_kb":312372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"74'178","reported_seq":1972,"reported_epoch":74,"state":"active+clean","last_fresh":"2023-10-03T17:48:46.006301+0000","last_change":"2023-10-03T17:40:17.641493+0000","last_active":"2023-10-03T17:48:46.006301+0000","last_peered":"2023-10-03T17:48:46.006301+0000","last_clean":"2023-10-03T17:48:46.006301+0000","last_became_active":"2023-10-03T17:34:42.278196+0000","last_became_peered":"2023-10-03T17:34:42.278196+0000","last_unstale":"2023-10-03T17:48:46.006301+0000","last_undegraded":"2023-10-03T17:48:46.006301+0000","last_fullsized":"2023-10-03T17:48:46.006301+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'30","last_scrub_stamp":"2023-10-03T17:40:17.640969+0000","last_deep_scrub":"46'30","last_deep_scrub_stamp":"2023-10-03T17:40:17.640969+0000","last_clean_scrub_stamp":"2023-10-03T17:40:17.640969+0000","objects_scrubbed":6,"log_size":178,"log_dups_size":0,"ondisk_log_size":178,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:04:02.954896+0000","scrub_duration":0.071054565,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":470,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":5125,"num_read_kb":4234,"num_write":178,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":903,"num_omap_keys":30,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"74'100486","reported_seq":168678,"reported_epoch":74,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T17:48:46.866093+0000","last_change":"2023-10-03T17:48:44.855815+0000","last_active":"2023-10-03T17:48:46.866093+0000","last_peered":"2023-10-03T17:48:46.866093+0000","last_clean":"2023-10-03T17:48:46.866093+0000","last_became_active":"2023-10-03T17:34:48.310912+0000","last_became_peered":"2023-10-03T17:34:48.310912+0000","last_unstale":"2023-10-03T17:48:46.866093+0000","last_undegraded":"2023-10-03T17:48:46.866093+0000","last_fullsized":"2023-10-03T17:48:46.866093+0000","mapping_epoch":16,"log_start":"70'90427","ondisk_log_start":"70'90427","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'58196","last_scrub_stamp":"2023-10-03T17:43:26.193075+0000","last_deep_scrub":"53'58196","last_deep_scrub_stamp":"2023-10-03T17:43:26.193075+0000","last_clean_scrub_stamp":"2023-10-03T17:43:26.193075+0000","objects_scrubbed":2326,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":7,"last_scrub_duration":196,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T21:42:26.771584+0000","scrub_duration":195.44739730800001,"objects_trimmed":801,"snaptrim_duration":41.465945443999999,"stat_sum":{"num_bytes":5415272448,"num_objects":6619,"num_object_clones":5348,"num_object_copies":13238,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6619,"num_whiteouts":0,"num_read":79739,"num_read_kb":300656,"num_write":80397,"num_write_kb":321588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"54","length":"1e"},{"start":"7c","length":"8"},{"start":"8a","length":"2"},{"start":"90","length":"a"},{"start":"a4","length":"a"},{"start":"b8","length":"a"},{"start":"cc","length":"3"}]},{"pgid":"2.0","version":"73'95","reported_seq":777,"reported_epoch":74,"state":"active+clean","last_fresh":"2023-10-03T17:48:46.020832+0000","last_change":"2023-10-03T17:40:10.708344+0000","last_active":"2023-10-03T17:48:46.020832+0000","last_peered":"2023-10-03T17:48:46.020832+0000","last_clean":"2023-10-03T17:48:46.020832+0000","last_became_active":"2023-10-03T17:34:42.272306+0000","last_became_peered":"2023-10-03T17:34:42.272306+0000","last_unstale":"2023-10-03T17:48:46.020832+0000","last_undegraded":"2023-10-03T17:48:46.020832+0000","last_fullsized":"2023-10-03T17:48:46.020832+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'20","last_scrub_stamp":"2023-10-03T17:40:10.708060+0000","last_deep_scrub":"46'20","last_deep_scrub_stamp":"2023-10-03T17:40:10.708060+0000","last_clean_scrub_stamp":"2023-10-03T17:40:10.708060+0000","objects_scrubbed":7,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:03:09.097252+0000","scrub_duration":0.042279491000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":806,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":1751,"num_read_kb":1430,"num_write":90,"num_write_kb":72,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":301,"num_omap_keys":10,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"74'100882","reported_seq":169824,"reported_epoch":74,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T17:48:47.041758+0000","last_change":"2023-10-03T17:48:45.588915+0000","last_active":"2023-10-03T17:48:47.041758+0000","last_peered":"2023-10-03T17:48:47.041758+0000","last_clean":"2023-10-03T17:48:47.041758+0000","last_became_active":"2023-10-03T17:34:48.310612+0000","last_became_peered":"2023-10-03T17:34:48.310612+0000","last_unstale":"2023-10-03T17:48:47.041758+0000","last_undegraded":"2023-10-03T17:48:47.041758+0000","last_fullsized":"2023-10-03T17:48:47.041758+0000","mapping_epoch":16,"log_start":"70'90807","ondisk_log_start":"70'90807","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'75807","last_scrub_stamp":"2023-10-03T17:45:43.833310+0000","last_deep_scrub":"61'75807","last_deep_scrub_stamp":"2023-10-03T17:45:43.833310+0000","last_clean_scrub_stamp":"2023-10-03T17:45:43.833310+0000","objects_scrubbed":3607,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":24,"last_scrub_duration":315,"scrub_schedule":"queued for deep scrub","scrub_duration":314.26374841099999,"objects_trimmed":724,"snaptrim_duration":42.228851548999998,"stat_sum":{"num_bytes":5483843603,"num_objects":7798,"num_object_clones":6510,"num_object_copies":15596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7798,"num_whiteouts":0,"num_read":80118,"num_read_kb":301484,"num_write":81456,"num_write_kb":325818,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"54","length":"1e"},{"start":"7c","length":"8"},{"start":"8a","length":"2"},{"start":"90","length":"a"},{"start":"a4","length":"6"}]},{"pgid":"2.3","version":"73'40","reported_seq":152,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-03T17:48:03.064718+0000","last_change":"2023-10-03T17:40:24.472734+0000","last_active":"2023-10-03T17:48:03.064718+0000","last_peered":"2023-10-03T17:48:03.064718+0000","last_clean":"2023-10-03T17:48:03.064718+0000","last_became_active":"2023-10-03T17:34:42.270194+0000","last_became_peered":"2023-10-03T17:34:42.270194+0000","last_unstale":"2023-10-03T17:48:03.064718+0000","last_undegraded":"2023-10-03T17:48:03.064718+0000","last_fullsized":"2023-10-03T17:48:03.064718+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'8","last_scrub_stamp":"2023-10-03T17:40:24.472573+0000","last_deep_scrub":"46'8","last_deep_scrub_stamp":"2023-10-03T17:40:24.472573+0000","last_clean_scrub_stamp":"2023-10-03T17:40:24.472573+0000","objects_scrubbed":4,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T18:28:04.735033+0000","scrub_duration":0.024990637999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":39,"num_read_kb":21,"num_write":36,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"46'99","reported_seq":218,"reported_epoch":74,"state":"active+clean","last_fresh":"2023-10-03T17:48:03.362081+0000","last_change":"2023-10-03T17:40:10.452341+0000","last_active":"2023-10-03T17:48:03.362081+0000","last_peered":"2023-10-03T17:48:03.362081+0000","last_clean":"2023-10-03T17:48:03.362081+0000","last_became_active":"2023-10-03T17:34:39.502393+0000","last_became_peered":"2023-10-03T17:34:39.502393+0000","last_unstale":"2023-10-03T17:48:03.362081+0000","last_undegraded":"2023-10-03T17:48:03.362081+0000","last_fullsized":"2023-10-03T17:48:03.362081+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'99","last_scrub_stamp":"2023-10-03T17:40:10.449099+0000","last_deep_scrub":"46'99","last_deep_scrub_stamp":"2023-10-03T17:40:10.449099+0000","last_clean_scrub_stamp":"2023-10-03T17:40:10.449099+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T20:18:04.958666+0000","scrub_duration":0.043225012,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":21811437587,"num_objects":34132,"num_object_clones":29011,"num_object_copies":68264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34132,"num_whiteouts":0,"num_read":319922,"num_read_kb":1203128,"num_write":323632,"num_write_kb":1294522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2651492352,"data_stored":13388054566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40213,"ondisk_log_size":40213,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":5372,"num_objects":143,"num_object_clones":0,"num_object_copies":286,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":143,"num_whiteouts":0,"num_read":36827,"num_read_kb":30755,"num_write":1403,"num_write_kb":1163,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":983040,"data_stored":10744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7029,"internal_metadata":0},"log_size":9585,"ondisk_log_size":9585,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":7460,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738544,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1957824,"kb_used_data":1295544,"kb_used_omap":2891,"kb_used_meta":659380,"kb_avail":92414016,"statfs":{"total":96636764160,"available":94631952384,"internally_reserved":0,"allocated":1326637056,"data_stored":6693272551,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2960945,"internal_metadata":675205583},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,4,2,1,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 17:48:01 2023","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.42199999999999999,"15min":0.45200000000000001},"min":{"1min":0.17199999999999999,"5min":0.16500000000000001,"15min":0.14299999999999999},"max":{"1min":3.9950000000000001,"5min":3.9950000000000001,"15min":4.4640000000000004},"last":0.28899999999999998},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.379,"15min":0.442},"min":{"1min":0.186,"5min":0.17799999999999999,"15min":0.17100000000000001},"max":{"1min":2.9049999999999998,"5min":2.9049999999999998,"15min":4.4180000000000001},"last":0.40699999999999997}]},{"osd":1,"last update":"Tue Oct 3 17:48:01 2023","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.54700000000000004,"15min":0.51900000000000002},"min":{"1min":0.222,"5min":0.184,"15min":0.16600000000000001},"max":{"1min":4.0359999999999996,"5min":8.1059999999999999,"15min":8.1059999999999999},"last":0.25600000000000001},{"interface":"front","average":{"1min":0.42599999999999999,"5min":0.46700000000000003,"15min":0.47199999999999998},"min":{"1min":0.23599999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.92400000000000004,"5min":4.1120000000000001,"15min":4.1120000000000001},"last":0.36699999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738542,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1673588,"kb_used_data":638828,"kb_used_omap":1,"kb_used_meta":1034750,"kb_avail":92698252,"statfs":{"total":96636764160,"available":94923010048,"internally_reserved":0,"allocated":654159872,"data_stored":3693833299,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":1059584454},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,7,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 17:48:14 2023","interfaces":[{"interface":"back","average":{"1min":0.32500000000000001,"5min":0.32600000000000001,"15min":0.371},"min":{"1min":0.17999999999999999,"5min":0.17199999999999999,"15min":0.161},"max":{"1min":1.895,"5min":2.3809999999999998,"15min":4.1879999999999997},"last":2.7050000000000001},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.47699999999999998,"15min":0.42999999999999999},"min":{"1min":0.191,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":5.8959999999999999,"5min":5.8959999999999999,"15min":5.8959999999999999},"last":0.30599999999999999}]},{"osd":2,"last update":"Tue Oct 3 17:48:14 2023","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.441,"15min":0.42299999999999999},"min":{"1min":0.184,"5min":0.184,"15min":0.159},"max":{"1min":2.6240000000000001,"5min":2.6240000000000001,"15min":3.0390000000000001},"last":0.24299999999999999},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.39700000000000002,"15min":0.41199999999999998},"min":{"1min":0.23200000000000001,"5min":0.16400000000000001,"15min":0.16400000000000001},"max":{"1min":1.9330000000000001,"5min":2.6970000000000001,"15min":3.0699999999999998},"last":2.665}]}]},{"osd":0,"up_from":8,"seq":34359738542,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1654488,"kb_used_data":657104,"kb_used_omap":1,"kb_used_meta":997374,"kb_avail":92717352,"statfs":{"total":96636764160,"available":94942568448,"internally_reserved":0,"allocated":672874496,"data_stored":3005375888,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":1021311430},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,0,5,8,8,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 17:48:00 2023","interfaces":[{"interface":"back","average":{"1min":0.318,"5min":0.373,"15min":0.41899999999999998},"min":{"1min":0.185,"5min":0.17100000000000001,"15min":0.154},"max":{"1min":0.83399999999999996,"5min":3.4649999999999999,"15min":5.7649999999999997},"last":0.251},{"interface":"front","average":{"1min":0.34699999999999998,"5min":0.38300000000000001,"15min":0.46200000000000002},"min":{"1min":0.20999999999999999,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":0.92600000000000005,"5min":3.4279999999999999,"15min":9.4770000000000003},"last":0.23699999999999999}]},{"osd":2,"last update":"Tue Oct 3 17:48:00 2023","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.53200000000000003,"15min":0.52600000000000002},"min":{"1min":0.216,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.95799999999999996,"5min":4.1619999999999999,"15min":9.4149999999999991},"last":0.26800000000000002},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.61099999999999999,"15min":0.50900000000000001},"min":{"1min":0.246,"5min":0.19700000000000001,"15min":0.189},"max":{"1min":0.89900000000000002,"5min":5.218,"15min":5.218},"last":0.27800000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":335872,"data_stored":3660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":5372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":155648,"data_stored":1712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7029,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":672092160,"data_stored":3002830848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":653512704,"data_stored":3692199955,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1325887488,"data_stored":6693023763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-03T17:48:49.890 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-03T17:34:47.281095+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=34, tm_sec=47, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=40, tm_sec=6, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T17:48:49.891 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T17:49:00.405 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:49:00.404+0000 7f09e4851640 -1 reset not still connected to 0x557d7cb140d0 2023-10-03T17:49:00.406 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:49:00.404+0000 7f09e4851640 -1 reset not still connected to 0x557d82f8af70 2023-10-03T17:49:00.406 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:49:00.404+0000 7f09e4851640 -1 reset not still connected to 0x557d8314a680 2023-10-03T17:49:00.407 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:49:00.404+0000 7f09e4851640 -1 reset not still connected to 0x557d85583ee0 2023-10-03T17:49:09.892 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T17:49:10.340 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-03T17:49:10.340 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-10-03T17:49:10.382 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":514,"stamp":"2023-10-03T17:49:10.154358+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":21844105007,"num_objects":34601,"num_object_clones":29335,"num_object_copies":69202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34601,"num_whiteouts":0,"num_read":365864,"num_read_kb":1259887,"num_write":331690,"num_write_kb":1322931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":50273,"ondisk_log_size":50273,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":5472296,"kb_used_data":2641872,"kb_used_omap":2894,"kb_used_meta":2827505,"kb_avail":277643224,"statfs":{"total":289910292480,"available":284306661376,"internally_reserved":0,"allocated":2705276928,"data_stored":13659767289,"data_compressed":7460,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":2964133,"internal_metadata":2895365467},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,7,6,8,16,14,31,8,1,3],"upper_bound":2048},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":7118848,"num_objects":818,"num_object_clones":818,"num_object_copies":1636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":818,"num_whiteouts":0,"num_read":2212,"num_read_kb":5856,"num_write":1401,"num_write_kb":5244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.801430"},"pg_stats":[{"pgid":"2.7","version":"79'1785","reported_seq":3716,"reported_epoch":79,"state":"active+clean","last_fresh":"2023-10-03T17:49:06.486454+0000","last_change":"2023-10-03T17:40:17.407011+0000","last_active":"2023-10-03T17:49:06.486454+0000","last_peered":"2023-10-03T17:49:06.486454+0000","last_clean":"2023-10-03T17:49:06.486454+0000","last_became_active":"2023-10-03T17:34:42.276624+0000","last_became_peered":"2023-10-03T17:34:42.276624+0000","last_unstale":"2023-10-03T17:49:06.486454+0000","last_undegraded":"2023-10-03T17:49:06.486454+0000","last_fullsized":"2023-10-03T17:49:06.486454+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'673","last_scrub_stamp":"2023-10-03T17:40:17.406813+0000","last_deep_scrub":"46'673","last_deep_scrub_stamp":"2023-10-03T17:40:17.406813+0000","last_clean_scrub_stamp":"2023-10-03T17:40:17.406813+0000","objects_scrubbed":7,"log_size":1785,"log_dups_size":0,"ondisk_log_size":1785,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T21:38:35.056889+0000","scrub_duration":0.013778076,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":639,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":5584,"num_read_kb":4612,"num_write":229,"num_write_kb":183,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":903,"num_omap_keys":30,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"79'2966","reported_seq":7432,"reported_epoch":79,"state":"active+clean","last_fresh":"2023-10-03T17:49:06.487457+0000","last_change":"2023-10-03T17:40:13.376519+0000","last_active":"2023-10-03T17:49:06.487457+0000","last_peered":"2023-10-03T17:49:06.487457+0000","last_clean":"2023-10-03T17:49:06.487457+0000","last_became_active":"2023-10-03T17:34:42.271201+0000","last_became_peered":"2023-10-03T17:34:42.271201+0000","last_unstale":"2023-10-03T17:49:06.487457+0000","last_undegraded":"2023-10-03T17:49:06.487457+0000","last_fullsized":"2023-10-03T17:49:06.487457+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'1129","last_scrub_stamp":"2023-10-03T17:40:13.376332+0000","last_deep_scrub":"46'1129","last_deep_scrub_stamp":"2023-10-03T17:40:13.376332+0000","last_clean_scrub_stamp":"2023-10-03T17:40:13.376332+0000","objects_scrubbed":11,"log_size":2966,"log_dups_size":0,"ondisk_log_size":2966,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T18:59:17.088585+0000","scrub_duration":0.015998675,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":673,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":11116,"num_read_kb":9504,"num_write":396,"num_write_kb":329,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2912,"num_omap_keys":93,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"78'47","reported_seq":167,"reported_epoch":78,"state":"active+clean","last_fresh":"2023-10-03T17:49:03.391577+0000","last_change":"2023-10-03T17:40:25.496010+0000","last_active":"2023-10-03T17:49:03.391577+0000","last_peered":"2023-10-03T17:49:03.391577+0000","last_clean":"2023-10-03T17:49:03.391577+0000","last_became_active":"2023-10-03T17:34:42.276941+0000","last_became_peered":"2023-10-03T17:34:42.276941+0000","last_unstale":"2023-10-03T17:49:03.391577+0000","last_undegraded":"2023-10-03T17:49:03.391577+0000","last_fullsized":"2023-10-03T17:49:03.391577+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'14","last_scrub_stamp":"2023-10-03T17:40:25.495827+0000","last_deep_scrub":"46'14","last_deep_scrub_stamp":"2023-10-03T17:40:25.495827+0000","last_clean_scrub_stamp":"2023-10-03T17:40:25.495827+0000","objects_scrubbed":7,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:05:44.939663+0000","scrub_duration":0.0133078,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1006,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":48,"num_read_kb":21,"num_write":40,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"79'2365","reported_seq":5582,"reported_epoch":79,"state":"active+clean","last_fresh":"2023-10-03T17:49:06.486701+0000","last_change":"2023-10-03T17:40:26.513902+0000","last_active":"2023-10-03T17:49:06.486701+0000","last_peered":"2023-10-03T17:49:06.486701+0000","last_clean":"2023-10-03T17:49:06.486701+0000","last_became_active":"2023-10-03T17:34:42.276459+0000","last_became_peered":"2023-10-03T17:34:42.276459+0000","last_unstale":"2023-10-03T17:49:06.486701+0000","last_undegraded":"2023-10-03T17:49:06.486701+0000","last_fullsized":"2023-10-03T17:49:06.486701+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'964","last_scrub_stamp":"2023-10-03T17:40:26.513729+0000","last_deep_scrub":"46'964","last_deep_scrub_stamp":"2023-10-03T17:40:26.513729+0000","last_clean_scrub_stamp":"2023-10-03T17:40:26.513729+0000","objects_scrubbed":10,"log_size":2365,"log_dups_size":0,"ondisk_log_size":2365,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:41:33.182751+0000","scrub_duration":0.018190280999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":333,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":8183,"num_read_kb":6842,"num_write":286,"num_write_kb":240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1731,"num_omap_keys":80,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"79'104426","reported_seq":176725,"reported_epoch":79,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T17:49:07.034260+0000","last_change":"2023-10-03T17:49:05.689843+0000","last_active":"2023-10-03T17:49:07.034260+0000","last_peered":"2023-10-03T17:49:07.034260+0000","last_clean":"2023-10-03T17:49:07.034260+0000","last_became_active":"2023-10-03T17:34:48.310701+0000","last_became_peered":"2023-10-03T17:34:48.310701+0000","last_unstale":"2023-10-03T17:49:07.034260+0000","last_undegraded":"2023-10-03T17:49:07.034260+0000","last_fullsized":"2023-10-03T17:49:07.034260+0000","mapping_epoch":16,"log_start":"70'94388","ondisk_log_start":"70'94388","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'80695","last_scrub_stamp":"2023-10-03T17:46:13.982571+0000","last_deep_scrub":"66'80695","last_deep_scrub_stamp":"2023-10-03T17:46:13.982571+0000","last_clean_scrub_stamp":"2023-10-03T17:46:13.982571+0000","objects_scrubbed":3410,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":51,"last_scrub_duration":363,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T23:51:11.410881+0000","scrub_duration":362.26574615599998,"objects_trimmed":44,"snaptrim_duration":1.2948399770000001,"stat_sum":{"num_bytes":5629657088,"num_objects":8804,"num_object_clones":7484,"num_object_copies":17608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8804,"num_whiteouts":0,"num_read":83937,"num_read_kb":315772,"num_write":85305,"num_write_kb":341220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"54","length":"1e"},{"start":"7c","length":"8"},{"start":"8a","length":"2"},{"start":"90","length":"9"}]},{"pgid":"2.2","version":"79'2454","reported_seq":5026,"reported_epoch":79,"state":"active+clean","last_fresh":"2023-10-03T17:49:06.486451+0000","last_change":"2023-10-03T17:40:08.675119+0000","last_active":"2023-10-03T17:49:06.486451+0000","last_peered":"2023-10-03T17:49:06.486451+0000","last_clean":"2023-10-03T17:49:06.486451+0000","last_became_active":"2023-10-03T17:34:42.270625+0000","last_became_peered":"2023-10-03T17:34:42.270625+0000","last_unstale":"2023-10-03T17:49:06.486451+0000","last_undegraded":"2023-10-03T17:49:06.486451+0000","last_fullsized":"2023-10-03T17:49:06.486451+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'841","last_scrub_stamp":"2023-10-03T17:40:08.674910+0000","last_deep_scrub":"46'841","last_deep_scrub_stamp":"2023-10-03T17:40:08.674910+0000","last_clean_scrub_stamp":"2023-10-03T17:40:08.674910+0000","objects_scrubbed":11,"log_size":2454,"log_dups_size":0,"ondisk_log_size":2454,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T21:05:42.790614+0000","scrub_duration":0.039189555000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":858,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":7465,"num_read_kb":6178,"num_write":317,"num_write_kb":260,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2564,"num_omap_keys":60,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"79'94175","reported_seq":163730,"reported_epoch":79,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T17:49:07.190298+0000","last_change":"2023-10-03T17:43:28.494717+0000","last_active":"2023-10-03T17:49:07.190298+0000","last_peered":"2023-10-03T17:49:07.190298+0000","last_clean":"2023-10-03T17:49:07.190298+0000","last_became_active":"2023-10-03T17:34:48.310555+0000","last_became_peered":"2023-10-03T17:34:48.310555+0000","last_unstale":"2023-10-03T17:49:07.190298+0000","last_undegraded":"2023-10-03T17:49:07.190298+0000","last_fullsized":"2023-10-03T17:49:07.190298+0000","mapping_epoch":16,"log_start":"70'84097","ondisk_log_start":"70'84097","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_clean_scrub_stamp":"2023-10-03T17:34:47.281095+0000","objects_scrubbed":5304,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":100,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 338s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086289699999999995,"stat_sum":{"num_bytes":5298589696,"num_objects":11314,"num_object_clones":10072,"num_object_copies":22628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11314,"num_whiteouts":0,"num_read":79149,"num_read_kb":297080,"num_write":79627,"num_write_kb":318508,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"79'204","reported_seq":2117,"reported_epoch":79,"state":"active+clean","last_fresh":"2023-10-03T17:49:07.168468+0000","last_change":"2023-10-03T17:40:17.641493+0000","last_active":"2023-10-03T17:49:07.168468+0000","last_peered":"2023-10-03T17:49:07.168468+0000","last_clean":"2023-10-03T17:49:07.168468+0000","last_became_active":"2023-10-03T17:34:42.278196+0000","last_became_peered":"2023-10-03T17:34:42.278196+0000","last_unstale":"2023-10-03T17:49:07.168468+0000","last_undegraded":"2023-10-03T17:49:07.168468+0000","last_fullsized":"2023-10-03T17:49:07.168468+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'30","last_scrub_stamp":"2023-10-03T17:40:17.640969+0000","last_deep_scrub":"46'30","last_deep_scrub_stamp":"2023-10-03T17:40:17.640969+0000","last_clean_scrub_stamp":"2023-10-03T17:40:17.640969+0000","objects_scrubbed":6,"log_size":204,"log_dups_size":0,"ondisk_log_size":204,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:04:02.954896+0000","scrub_duration":0.071054565,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":521,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":5556,"num_read_kb":4593,"num_write":210,"num_write_kb":172,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":903,"num_omap_keys":30,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"79'103221","reported_seq":172296,"reported_epoch":79,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T17:49:07.021953+0000","last_change":"2023-10-03T17:49:05.297963+0000","last_active":"2023-10-03T17:49:07.021953+0000","last_peered":"2023-10-03T17:49:07.021953+0000","last_clean":"2023-10-03T17:49:07.021953+0000","last_became_active":"2023-10-03T17:34:48.310912+0000","last_became_peered":"2023-10-03T17:34:48.310912+0000","last_unstale":"2023-10-03T17:49:07.021953+0000","last_undegraded":"2023-10-03T17:49:07.021953+0000","last_fullsized":"2023-10-03T17:49:07.021953+0000","mapping_epoch":16,"log_start":"70'93140","ondisk_log_start":"70'93140","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'58196","last_scrub_stamp":"2023-10-03T17:43:26.193075+0000","last_deep_scrub":"53'58196","last_deep_scrub_stamp":"2023-10-03T17:43:26.193075+0000","last_clean_scrub_stamp":"2023-10-03T17:43:26.193075+0000","objects_scrubbed":2326,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":20,"last_scrub_duration":196,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:00:09.852937+0000","scrub_duration":195.44739730800001,"objects_trimmed":40,"snaptrim_duration":0.91176273399999996,"stat_sum":{"num_bytes":5423001600,"num_objects":6566,"num_object_clones":5295,"num_object_copies":13132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6566,"num_whiteouts":0,"num_read":81150,"num_read_kb":306188,"num_write":81956,"num_write_kb":327824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"54","length":"1e"},{"start":"7c","length":"8"},{"start":"8a","length":"2"},{"start":"90","length":"a"},{"start":"a4","length":"a"},{"start":"b8","length":"a"},{"start":"cc","length":"a"}]},{"pgid":"2.0","version":"78'108","reported_seq":838,"reported_epoch":79,"state":"active+clean","last_fresh":"2023-10-03T17:49:04.391882+0000","last_change":"2023-10-03T17:40:10.708344+0000","last_active":"2023-10-03T17:49:04.391882+0000","last_peered":"2023-10-03T17:49:04.391882+0000","last_clean":"2023-10-03T17:49:04.391882+0000","last_became_active":"2023-10-03T17:34:42.272306+0000","last_became_peered":"2023-10-03T17:34:42.272306+0000","last_unstale":"2023-10-03T17:49:04.391882+0000","last_undegraded":"2023-10-03T17:49:04.391882+0000","last_fullsized":"2023-10-03T17:49:04.391882+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'20","last_scrub_stamp":"2023-10-03T17:40:10.708060+0000","last_deep_scrub":"46'20","last_deep_scrub_stamp":"2023-10-03T17:40:10.708060+0000","last_clean_scrub_stamp":"2023-10-03T17:40:10.708060+0000","objects_scrubbed":7,"log_size":108,"log_dups_size":0,"ondisk_log_size":108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:03:09.097252+0000","scrub_duration":0.042279491000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":805,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":1901,"num_read_kb":1556,"num_write":105,"num_write_kb":82,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":301,"num_omap_keys":10,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"79'103624","reported_seq":173543,"reported_epoch":79,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T17:49:07.200584+0000","last_change":"2023-10-03T17:49:03.945100+0000","last_active":"2023-10-03T17:49:07.200584+0000","last_peered":"2023-10-03T17:49:07.200584+0000","last_clean":"2023-10-03T17:49:07.200584+0000","last_became_active":"2023-10-03T17:34:48.310612+0000","last_became_peered":"2023-10-03T17:34:48.310612+0000","last_unstale":"2023-10-03T17:49:07.200584+0000","last_undegraded":"2023-10-03T17:49:07.200584+0000","last_fullsized":"2023-10-03T17:49:07.200584+0000","mapping_epoch":16,"log_start":"70'93620","ondisk_log_start":"70'93620","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'75807","last_scrub_stamp":"2023-10-03T17:45:43.833310+0000","last_deep_scrub":"61'75807","last_deep_scrub_stamp":"2023-10-03T17:45:43.833310+0000","last_clean_scrub_stamp":"2023-10-03T17:45:43.833310+0000","objects_scrubbed":3607,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":38,"last_scrub_duration":315,"scrub_schedule":"queued for deep scrub","scrub_duration":314.26374841099999,"objects_trimmed":38,"snaptrim_duration":0.56886025699999998,"stat_sum":{"num_bytes":5492260883,"num_objects":7772,"num_object_clones":6484,"num_object_copies":15544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7772,"num_whiteouts":0,"num_read":81629,"num_read_kb":307428,"num_write":83042,"num_write_kb":332162,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"54","length":"1e"},{"start":"7c","length":"8"},{"start":"8a","length":"2"},{"start":"90","length":"a"},{"start":"a4","length":"a"},{"start":"b8","length":"2"}]},{"pgid":"2.3","version":"77'44","reported_seq":162,"reported_epoch":78,"state":"active+clean","last_fresh":"2023-10-03T17:49:03.359096+0000","last_change":"2023-10-03T17:40:24.472734+0000","last_active":"2023-10-03T17:49:03.359096+0000","last_peered":"2023-10-03T17:49:03.359096+0000","last_clean":"2023-10-03T17:49:03.359096+0000","last_became_active":"2023-10-03T17:34:42.270194+0000","last_became_peered":"2023-10-03T17:34:42.270194+0000","last_unstale":"2023-10-03T17:49:03.359096+0000","last_undegraded":"2023-10-03T17:49:03.359096+0000","last_fullsized":"2023-10-03T17:49:03.359096+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'8","last_scrub_stamp":"2023-10-03T17:40:24.472573+0000","last_deep_scrub":"46'8","last_deep_scrub_stamp":"2023-10-03T17:40:24.472573+0000","last_clean_scrub_stamp":"2023-10-03T17:40:24.472573+0000","objects_scrubbed":4,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T18:28:04.735033+0000","scrub_duration":0.024990637999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":537,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":40,"num_read_kb":22,"num_write":40,"num_write_kb":26,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"46'99","reported_seq":223,"reported_epoch":78,"state":"active+clean","last_fresh":"2023-10-03T17:49:03.376720+0000","last_change":"2023-10-03T17:40:10.452341+0000","last_active":"2023-10-03T17:49:03.376720+0000","last_peered":"2023-10-03T17:49:03.376720+0000","last_clean":"2023-10-03T17:49:03.376720+0000","last_became_active":"2023-10-03T17:34:39.502393+0000","last_became_peered":"2023-10-03T17:34:39.502393+0000","last_unstale":"2023-10-03T17:49:03.376720+0000","last_undegraded":"2023-10-03T17:49:03.376720+0000","last_fullsized":"2023-10-03T17:49:03.376720+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'99","last_scrub_stamp":"2023-10-03T17:40:10.449099+0000","last_deep_scrub":"46'99","last_deep_scrub_stamp":"2023-10-03T17:40:10.449099+0000","last_clean_scrub_stamp":"2023-10-03T17:40:10.449099+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T20:18:04.958666+0000","scrub_duration":0.043225012,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":21843509267,"num_objects":34456,"num_object_clones":29335,"num_object_copies":68912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34456,"num_whiteouts":0,"num_read":325865,"num_read_kb":1226468,"num_write":329930,"num_write_kb":1319714,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2703011840,"data_stored":13690097702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40201,"ondisk_log_size":40201,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":5372,"num_objects":143,"num_object_clones":0,"num_object_copies":286,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":143,"num_whiteouts":0,"num_read":39893,"num_read_kb":33328,"num_write":1623,"num_write_kb":1323,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":983040,"data_stored":10744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7029,"internal_metadata":0},"log_size":9973,"ondisk_log_size":9973,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":7460,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738548,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2046820,"kb_used_data":1320668,"kb_used_omap":2891,"kb_used_meta":723252,"kb_avail":92325020,"statfs":{"total":96636764160,"available":94540820480,"internally_reserved":0,"allocated":1352364032,"data_stored":6846434787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2960945,"internal_metadata":740610511},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,3,6,8,14,4,1,3],"upper_bound":2048},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 17:49:01 2023","interfaces":[{"interface":"back","average":{"1min":0.33300000000000002,"5min":0.41299999999999998,"15min":0.44},"min":{"1min":0.19600000000000001,"5min":0.16500000000000001,"15min":0.14299999999999999},"max":{"1min":0.751,"5min":3.9950000000000001,"15min":4.4640000000000004},"last":0.216},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.38300000000000001,"15min":0.42999999999999999},"min":{"1min":0.224,"5min":0.17999999999999999,"15min":0.17100000000000001},"max":{"1min":0.73399999999999999,"5min":2.9049999999999998,"15min":4.4180000000000001},"last":0.185}]},{"osd":1,"last update":"Tue Oct 3 17:49:01 2023","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.54500000000000004,"15min":0.50900000000000001},"min":{"1min":0.22800000000000001,"5min":0.184,"15min":0.16600000000000001},"max":{"1min":1.9390000000000001,"5min":8.1059999999999999,"15min":8.1059999999999999},"last":0.253},{"interface":"front","average":{"1min":0.35699999999999998,"5min":0.46899999999999997,"15min":0.45600000000000002},"min":{"1min":0.22600000000000001,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.69999999999999996,"5min":4.1120000000000001,"15min":4.1120000000000001},"last":0.29399999999999998}]}]},{"osd":1,"up_from":8,"seq":34359738546,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1704244,"kb_used_data":651436,"kb_used_omap":1,"kb_used_meta":1052798,"kb_avail":92667596,"statfs":{"total":96636764160,"available":94891618304,"internally_reserved":0,"allocated":667070464,"data_stored":3786018232,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":1078065606},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,2,0,5,4,7,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 17:48:14 2023","interfaces":[{"interface":"back","average":{"1min":0.32500000000000001,"5min":0.32600000000000001,"15min":0.371},"min":{"1min":0.17999999999999999,"5min":0.17199999999999999,"15min":0.161},"max":{"1min":1.895,"5min":2.3809999999999998,"15min":4.1879999999999997},"last":0.185},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.47699999999999998,"15min":0.42999999999999999},"min":{"1min":0.191,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":5.8959999999999999,"5min":5.8959999999999999,"15min":5.8959999999999999},"last":0.23200000000000001}]},{"osd":2,"last update":"Tue Oct 3 17:48:14 2023","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.441,"15min":0.42299999999999999},"min":{"1min":0.184,"5min":0.184,"15min":0.159},"max":{"1min":2.6240000000000001,"5min":2.6240000000000001,"15min":3.0390000000000001},"last":0.17599999999999999},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.39700000000000002,"15min":0.41199999999999998},"min":{"1min":0.23200000000000001,"5min":0.16400000000000001,"15min":0.16400000000000001},"max":{"1min":1.9330000000000001,"5min":2.6970000000000001,"15min":3.0699999999999998},"last":0.32000000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738546,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1721232,"kb_used_data":669768,"kb_used_omap":1,"kb_used_meta":1051454,"kb_avail":92650608,"statfs":{"total":96636764160,"available":94874222592,"internally_reserved":0,"allocated":685842432,"data_stored":3027314270,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":1076689350},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,3,5,5,2,10,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 17:49:02 2023","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.39700000000000002,"15min":0.41499999999999998},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.154},"max":{"1min":1.579,"5min":3.4649999999999999,"15min":5.7649999999999997},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.502,"5min":0.42499999999999999,"15min":0.46300000000000002},"min":{"1min":0.23599999999999999,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":1.679,"5min":3.4279999999999999,"15min":9.4770000000000003},"last":0.44900000000000001}]},{"osd":2,"last update":"Tue Oct 3 17:49:02 2023","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.49399999999999999,"15min":0.51400000000000001},"min":{"1min":0.223,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":1.244,"5min":4.1619999999999999,"15min":9.4149999999999991},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.56799999999999995,"15min":0.495},"min":{"1min":0.224,"5min":0.19700000000000001,"15min":0.189},"max":{"1min":0.66900000000000004,"5min":5.218,"15min":5.218},"last":0.313}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":323584,"data_stored":3509,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":5372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":167936,"data_stored":1863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7029,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":685101056,"data_stored":3038269440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":666288128,"data_stored":3803123731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1351622656,"data_stored":6848704531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-03T17:49:10.384 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-03T17:34:47.281095+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=34, tm_sec=47, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=40, tm_sec=6, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T17:49:10.384 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T17:49:20.278 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:49:20.280+0000 7f09e4851640 -1 reset not still connected to 0x557d76253d40 2023-10-03T17:49:20.281 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:49:20.284+0000 7f09e4851640 -1 reset not still connected to 0x557d76621040 2023-10-03T17:49:20.281 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:49:20.284+0000 7f09e4851640 -1 reset not still connected to 0x557d7664e750 2023-10-03T17:49:20.281 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:49:20.284+0000 7f09e4851640 -1 reset not still connected to 0x557d7664ea90 2023-10-03T17:49:20.282 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:49:20.284+0000 7f09e4851640 -1 reset not still connected to 0x557d7664f380 2023-10-03T17:49:20.285 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:49:20.288+0000 7f09e4851640 -1 reset not still connected to 0x557d7664f930 2023-10-03T17:49:20.285 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:49:20.288+0000 7f09e4851640 -1 reset not still connected to 0x557d76681ba0 2023-10-03T17:49:20.286 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:49:20.288+0000 7f09e4851640 -1 reset not still connected to 0x557d766a40d0 2023-10-03T17:49:20.286 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:49:20.288+0000 7f09e4851640 -1 reset not still connected to 0x557d766b56c0 2023-10-03T17:49:20.286 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:49:20.288+0000 7f09e4851640 -1 reset not still connected to 0x557d76eca270 2023-10-03T17:49:20.286 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:49:20.288+0000 7f09e4851640 -1 reset not still connected to 0x557d76f93ba0 2023-10-03T17:49:22.175 INFO:tasks.ceph.osd.0.smithi111.stderr:2023-10-03T17:49:22.172+0000 7ffbac289640 -1 reset not still connected to 0x5563a7ff68f0 2023-10-03T17:49:22.176 INFO:tasks.ceph.osd.0.smithi111.stderr:2023-10-03T17:49:22.172+0000 7ffbac289640 -1 reset not still connected to 0x5563a838e1a0 2023-10-03T17:49:27.261 INFO:tasks.ceph.osd.0.smithi111.stderr:2023-10-03T17:49:27.264+0000 7ffbac289640 -1 reset not still connected to 0x5563a7ff65b0 2023-10-03T17:49:27.261 INFO:tasks.ceph.osd.0.smithi111.stderr:2023-10-03T17:49:27.264+0000 7ffbac289640 -1 reset not still connected to 0x5563a7ff68f0 2023-10-03T17:49:27.262 INFO:tasks.ceph.osd.0.smithi111.stderr:2023-10-03T17:49:27.264+0000 7ffbac289640 -1 reset not still connected to 0x5563a838e1a0 2023-10-03T17:49:27.262 INFO:tasks.ceph.osd.0.smithi111.stderr:2023-10-03T17:49:27.264+0000 7ffbac289640 -1 reset not still connected to 0x5563a851aea0 2023-10-03T17:49:30.002 INFO:tasks.ceph.mgr.x.smithi111.stderr:2023-10-03T17:49:30.004+0000 7fe48bb97640 -1 calc_pg_upmaps abort due to max <= 0 2023-10-03T17:49:30.385 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T17:49:30.914 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-03T17:49:30.915 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-10-03T17:49:30.956 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":524,"stamp":"2023-10-03T17:49:30.161387+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":21872555823,"num_objects":36135,"num_object_clones":30869,"num_object_copies":72270,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36135,"num_whiteouts":0,"num_read":372707,"num_read_kb":1282498,"num_write":337111,"num_write_kb":1344615,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":50270,"ondisk_log_size":50270,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":5667532,"kb_used_data":2685556,"kb_used_omap":2894,"kb_used_meta":2979057,"kb_avail":277447988,"statfs":{"total":289910292480,"available":284106739712,"internally_reserved":0,"allocated":2750009344,"data_stored":14583226873,"data_compressed":7460,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":2964133,"internal_metadata":3050554715},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,5,2,10,11,13,12,9],"upper_bound":512},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":14028800,"num_objects":476,"num_object_clones":476,"num_object_copies":952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":476,"num_whiteouts":0,"num_read":2877,"num_read_kb":11334,"num_write":2754,"num_write_kb":11016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002014"},"pg_stats":[{"pgid":"2.7","version":"79'1806","reported_seq":3782,"reported_epoch":79,"state":"active+clean","last_fresh":"2023-10-03T17:49:26.966348+0000","last_change":"2023-10-03T17:40:17.407011+0000","last_active":"2023-10-03T17:49:26.966348+0000","last_peered":"2023-10-03T17:49:26.966348+0000","last_clean":"2023-10-03T17:49:26.966348+0000","last_became_active":"2023-10-03T17:34:42.276624+0000","last_became_peered":"2023-10-03T17:34:42.276624+0000","last_unstale":"2023-10-03T17:49:26.966348+0000","last_undegraded":"2023-10-03T17:49:26.966348+0000","last_fullsized":"2023-10-03T17:49:26.966348+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'673","last_scrub_stamp":"2023-10-03T17:40:17.406813+0000","last_deep_scrub":"46'673","last_deep_scrub_stamp":"2023-10-03T17:40:17.406813+0000","last_clean_scrub_stamp":"2023-10-03T17:40:17.406813+0000","objects_scrubbed":7,"log_size":1806,"log_dups_size":0,"ondisk_log_size":1806,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T21:38:35.056889+0000","scrub_duration":0.013778076,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":639,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":5782,"num_read_kb":4774,"num_write":229,"num_write_kb":183,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":903,"num_omap_keys":30,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"79'3001","reported_seq":7584,"reported_epoch":79,"state":"active+clean","last_fresh":"2023-10-03T17:49:26.966957+0000","last_change":"2023-10-03T17:40:13.376519+0000","last_active":"2023-10-03T17:49:26.966957+0000","last_peered":"2023-10-03T17:49:26.966957+0000","last_clean":"2023-10-03T17:49:26.966957+0000","last_became_active":"2023-10-03T17:34:42.271201+0000","last_became_peered":"2023-10-03T17:34:42.271201+0000","last_unstale":"2023-10-03T17:49:26.966957+0000","last_undegraded":"2023-10-03T17:49:26.966957+0000","last_fullsized":"2023-10-03T17:49:26.966957+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'1129","last_scrub_stamp":"2023-10-03T17:40:13.376332+0000","last_deep_scrub":"46'1129","last_deep_scrub_stamp":"2023-10-03T17:40:13.376332+0000","last_clean_scrub_stamp":"2023-10-03T17:40:13.376332+0000","objects_scrubbed":11,"log_size":3001,"log_dups_size":0,"ondisk_log_size":3001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T18:59:17.088585+0000","scrub_duration":0.015998675,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":673,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":11530,"num_read_kb":9857,"num_write":396,"num_write_kb":329,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2912,"num_omap_keys":93,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"78'47","reported_seq":168,"reported_epoch":79,"state":"active+clean","last_fresh":"2023-10-03T17:49:04.392628+0000","last_change":"2023-10-03T17:40:25.496010+0000","last_active":"2023-10-03T17:49:04.392628+0000","last_peered":"2023-10-03T17:49:04.392628+0000","last_clean":"2023-10-03T17:49:04.392628+0000","last_became_active":"2023-10-03T17:34:42.276941+0000","last_became_peered":"2023-10-03T17:34:42.276941+0000","last_unstale":"2023-10-03T17:49:04.392628+0000","last_undegraded":"2023-10-03T17:49:04.392628+0000","last_fullsized":"2023-10-03T17:49:04.392628+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'14","last_scrub_stamp":"2023-10-03T17:40:25.495827+0000","last_deep_scrub":"46'14","last_deep_scrub_stamp":"2023-10-03T17:40:25.495827+0000","last_clean_scrub_stamp":"2023-10-03T17:40:25.495827+0000","objects_scrubbed":7,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:05:44.939663+0000","scrub_duration":0.0133078,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1006,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":48,"num_read_kb":21,"num_write":40,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"79'2393","reported_seq":5693,"reported_epoch":79,"state":"active+clean","last_fresh":"2023-10-03T17:49:26.966809+0000","last_change":"2023-10-03T17:40:26.513902+0000","last_active":"2023-10-03T17:49:26.966809+0000","last_peered":"2023-10-03T17:49:26.966809+0000","last_clean":"2023-10-03T17:49:26.966809+0000","last_became_active":"2023-10-03T17:34:42.276459+0000","last_became_peered":"2023-10-03T17:34:42.276459+0000","last_unstale":"2023-10-03T17:49:26.966809+0000","last_undegraded":"2023-10-03T17:49:26.966809+0000","last_fullsized":"2023-10-03T17:49:26.966809+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'964","last_scrub_stamp":"2023-10-03T17:40:26.513729+0000","last_deep_scrub":"46'964","last_deep_scrub_stamp":"2023-10-03T17:40:26.513729+0000","last_clean_scrub_stamp":"2023-10-03T17:40:26.513729+0000","objects_scrubbed":10,"log_size":2393,"log_dups_size":0,"ondisk_log_size":2393,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:41:33.182751+0000","scrub_duration":0.018190280999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":333,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":8469,"num_read_kb":7079,"num_write":286,"num_write_kb":240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1731,"num_omap_keys":80,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"79'107118","reported_seq":180075,"reported_epoch":79,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T17:49:27.230598+0000","last_change":"2023-10-03T17:49:23.286071+0000","last_active":"2023-10-03T17:49:27.230598+0000","last_peered":"2023-10-03T17:49:27.230598+0000","last_clean":"2023-10-03T17:49:27.230598+0000","last_became_active":"2023-10-03T17:34:48.310701+0000","last_became_peered":"2023-10-03T17:34:48.310701+0000","last_unstale":"2023-10-03T17:49:27.230598+0000","last_undegraded":"2023-10-03T17:49:27.230598+0000","last_fullsized":"2023-10-03T17:49:27.230598+0000","mapping_epoch":16,"log_start":"74'97101","ondisk_log_start":"74'97101","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'80695","last_scrub_stamp":"2023-10-03T17:46:13.982571+0000","last_deep_scrub":"66'80695","last_deep_scrub_stamp":"2023-10-03T17:46:13.982571+0000","last_clean_scrub_stamp":"2023-10-03T17:46:13.982571+0000","objects_scrubbed":3410,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":47,"last_scrub_duration":363,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T23:51:11.410881+0000","scrub_duration":362.26574615599998,"objects_trimmed":325,"snaptrim_duration":18.882182714999999,"stat_sum":{"num_bytes":5637419008,"num_objects":9180,"num_object_clones":7860,"num_object_copies":18360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9180,"num_whiteouts":0,"num_read":85370,"num_read_kb":321448,"num_write":86778,"num_write_kb":347112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"54","length":"1e"},{"start":"7c","length":"8"},{"start":"8a","length":"2"},{"start":"90","length":"a"},{"start":"a4","length":"3"}]},{"pgid":"2.2","version":"79'2482","reported_seq":5117,"reported_epoch":79,"state":"active+clean","last_fresh":"2023-10-03T17:49:26.966411+0000","last_change":"2023-10-03T17:40:08.675119+0000","last_active":"2023-10-03T17:49:26.966411+0000","last_peered":"2023-10-03T17:49:26.966411+0000","last_clean":"2023-10-03T17:49:26.966411+0000","last_became_active":"2023-10-03T17:34:42.270625+0000","last_became_peered":"2023-10-03T17:34:42.270625+0000","last_unstale":"2023-10-03T17:49:26.966411+0000","last_undegraded":"2023-10-03T17:49:26.966411+0000","last_fullsized":"2023-10-03T17:49:26.966411+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'841","last_scrub_stamp":"2023-10-03T17:40:08.674910+0000","last_deep_scrub":"46'841","last_deep_scrub_stamp":"2023-10-03T17:40:08.674910+0000","last_clean_scrub_stamp":"2023-10-03T17:40:08.674910+0000","objects_scrubbed":11,"log_size":2482,"log_dups_size":0,"ondisk_log_size":2482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T21:05:42.790614+0000","scrub_duration":0.039189555000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":858,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":7731,"num_read_kb":6397,"num_write":317,"num_write_kb":260,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2564,"num_omap_keys":60,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"79'96321","reported_seq":166583,"reported_epoch":79,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T17:49:27.344921+0000","last_change":"2023-10-03T17:43:28.494717+0000","last_active":"2023-10-03T17:49:27.344921+0000","last_peered":"2023-10-03T17:49:27.344921+0000","last_clean":"2023-10-03T17:49:27.344921+0000","last_became_active":"2023-10-03T17:34:48.310555+0000","last_became_peered":"2023-10-03T17:34:48.310555+0000","last_unstale":"2023-10-03T17:49:27.344921+0000","last_undegraded":"2023-10-03T17:49:27.344921+0000","last_fullsized":"2023-10-03T17:49:27.344921+0000","mapping_epoch":16,"log_start":"70'86303","ondisk_log_start":"70'86303","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_clean_scrub_stamp":"2023-10-03T17:34:47.281095+0000","objects_scrubbed":5627,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":100,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 358s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086289699999999995,"stat_sum":{"num_bytes":5304725504,"num_objects":11911,"num_object_clones":10669,"num_object_copies":23822,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11911,"num_whiteouts":0,"num_read":80430,"num_read_kb":302140,"num_write":80889,"num_write_kb":323556,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"79'204","reported_seq":2163,"reported_epoch":79,"state":"active+clean","last_fresh":"2023-10-03T17:49:26.966928+0000","last_change":"2023-10-03T17:40:17.641493+0000","last_active":"2023-10-03T17:49:26.966928+0000","last_peered":"2023-10-03T17:49:26.966928+0000","last_clean":"2023-10-03T17:49:26.966928+0000","last_became_active":"2023-10-03T17:34:42.278196+0000","last_became_peered":"2023-10-03T17:34:42.278196+0000","last_unstale":"2023-10-03T17:49:26.966928+0000","last_undegraded":"2023-10-03T17:49:26.966928+0000","last_fullsized":"2023-10-03T17:49:26.966928+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'30","last_scrub_stamp":"2023-10-03T17:40:17.640969+0000","last_deep_scrub":"46'30","last_deep_scrub_stamp":"2023-10-03T17:40:17.640969+0000","last_clean_scrub_stamp":"2023-10-03T17:40:17.640969+0000","objects_scrubbed":6,"log_size":204,"log_dups_size":0,"ondisk_log_size":204,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:04:02.954896+0000","scrub_duration":0.071054565,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":521,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":5754,"num_read_kb":4755,"num_write":210,"num_write_kb":172,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":903,"num_omap_keys":30,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"79'105863","reported_seq":175495,"reported_epoch":79,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T17:49:27.221612+0000","last_change":"2023-10-03T17:49:25.678379+0000","last_active":"2023-10-03T17:49:27.221612+0000","last_peered":"2023-10-03T17:49:27.221612+0000","last_clean":"2023-10-03T17:49:27.221612+0000","last_became_active":"2023-10-03T17:34:48.310912+0000","last_became_peered":"2023-10-03T17:34:48.310912+0000","last_unstale":"2023-10-03T17:49:27.221612+0000","last_undegraded":"2023-10-03T17:49:27.221612+0000","last_fullsized":"2023-10-03T17:49:27.221612+0000","mapping_epoch":16,"log_start":"74'95856","ondisk_log_start":"74'95856","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'58196","last_scrub_stamp":"2023-10-03T17:43:26.193075+0000","last_deep_scrub":"53'58196","last_deep_scrub_stamp":"2023-10-03T17:43:26.193075+0000","last_clean_scrub_stamp":"2023-10-03T17:43:26.193075+0000","objects_scrubbed":2326,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":15,"last_scrub_duration":196,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:00:09.852937+0000","scrub_duration":195.44739730800001,"objects_trimmed":379,"snaptrim_duration":21.291643955000001,"stat_sum":{"num_bytes":5430296576,"num_objects":6847,"num_object_clones":5576,"num_object_copies":13694,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6847,"num_whiteouts":0,"num_read":82465,"num_read_kb":311404,"num_write":83300,"num_write_kb":333200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"54","length":"1e"},{"start":"7c","length":"8"},{"start":"8a","length":"2"},{"start":"90","length":"a"},{"start":"a4","length":"a"},{"start":"b8","length":"a"},{"start":"cc","length":"a"},{"start":"e0","length":"5"}]},{"pgid":"2.0","version":"78'108","reported_seq":853,"reported_epoch":79,"state":"active+clean","last_fresh":"2023-10-03T17:49:21.848150+0000","last_change":"2023-10-03T17:40:10.708344+0000","last_active":"2023-10-03T17:49:21.848150+0000","last_peered":"2023-10-03T17:49:21.848150+0000","last_clean":"2023-10-03T17:49:21.848150+0000","last_became_active":"2023-10-03T17:34:42.272306+0000","last_became_peered":"2023-10-03T17:34:42.272306+0000","last_unstale":"2023-10-03T17:49:21.848150+0000","last_undegraded":"2023-10-03T17:49:21.848150+0000","last_fullsized":"2023-10-03T17:49:21.848150+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'20","last_scrub_stamp":"2023-10-03T17:40:10.708060+0000","last_deep_scrub":"46'20","last_deep_scrub_stamp":"2023-10-03T17:40:10.708060+0000","last_clean_scrub_stamp":"2023-10-03T17:40:10.708060+0000","objects_scrubbed":7,"log_size":108,"log_dups_size":0,"ondisk_log_size":108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:03:09.097252+0000","scrub_duration":0.042279491000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":805,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":1967,"num_read_kb":1610,"num_write":105,"num_write_kb":82,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":301,"num_omap_keys":10,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"79'106281","reported_seq":176820,"reported_epoch":79,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T17:49:27.377703+0000","last_change":"2023-10-03T17:49:26.384620+0000","last_active":"2023-10-03T17:49:27.377703+0000","last_peered":"2023-10-03T17:49:27.377703+0000","last_clean":"2023-10-03T17:49:27.377703+0000","last_became_active":"2023-10-03T17:34:48.310612+0000","last_became_peered":"2023-10-03T17:34:48.310612+0000","last_unstale":"2023-10-03T17:49:27.377703+0000","last_undegraded":"2023-10-03T17:49:27.377703+0000","last_fullsized":"2023-10-03T17:49:27.377703+0000","mapping_epoch":16,"log_start":"74'96237","ondisk_log_start":"74'96237","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'75807","last_scrub_stamp":"2023-10-03T17:45:43.833310+0000","last_deep_scrub":"61'75807","last_deep_scrub_stamp":"2023-10-03T17:45:43.833310+0000","last_clean_scrub_stamp":"2023-10-03T17:45:43.833310+0000","objects_scrubbed":3607,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":32,"last_scrub_duration":315,"scrub_schedule":"queued for deep scrub","scrub_duration":314.26374841099999,"objects_trimmed":383,"snaptrim_duration":21.973752317999999,"stat_sum":{"num_bytes":5499518995,"num_objects":8052,"num_object_clones":6764,"num_object_copies":16104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8052,"num_whiteouts":0,"num_read":83015,"num_read_kb":312900,"num_write":84384,"num_write_kb":337530,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"54","length":"1e"},{"start":"7c","length":"8"},{"start":"8a","length":"2"},{"start":"90","length":"a"},{"start":"a4","length":"a"},{"start":"b8","length":"8"}]},{"pgid":"2.3","version":"77'44","reported_seq":163,"reported_epoch":79,"state":"active+clean","last_fresh":"2023-10-03T17:49:04.392591+0000","last_change":"2023-10-03T17:40:24.472734+0000","last_active":"2023-10-03T17:49:04.392591+0000","last_peered":"2023-10-03T17:49:04.392591+0000","last_clean":"2023-10-03T17:49:04.392591+0000","last_became_active":"2023-10-03T17:34:42.270194+0000","last_became_peered":"2023-10-03T17:34:42.270194+0000","last_unstale":"2023-10-03T17:49:04.392591+0000","last_undegraded":"2023-10-03T17:49:04.392591+0000","last_fullsized":"2023-10-03T17:49:04.392591+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'8","last_scrub_stamp":"2023-10-03T17:40:24.472573+0000","last_deep_scrub":"46'8","last_deep_scrub_stamp":"2023-10-03T17:40:24.472573+0000","last_clean_scrub_stamp":"2023-10-03T17:40:24.472573+0000","objects_scrubbed":4,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T18:28:04.735033+0000","scrub_duration":0.024990637999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":537,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":40,"num_read_kb":22,"num_write":40,"num_write_kb":26,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"46'99","reported_seq":223,"reported_epoch":78,"state":"active+clean","last_fresh":"2023-10-03T17:49:03.376720+0000","last_change":"2023-10-03T17:40:10.452341+0000","last_active":"2023-10-03T17:49:03.376720+0000","last_peered":"2023-10-03T17:49:03.376720+0000","last_clean":"2023-10-03T17:49:03.376720+0000","last_became_active":"2023-10-03T17:34:39.502393+0000","last_became_peered":"2023-10-03T17:34:39.502393+0000","last_unstale":"2023-10-03T17:49:03.376720+0000","last_undegraded":"2023-10-03T17:49:03.376720+0000","last_fullsized":"2023-10-03T17:49:03.376720+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'99","last_scrub_stamp":"2023-10-03T17:40:10.449099+0000","last_deep_scrub":"46'99","last_deep_scrub_stamp":"2023-10-03T17:40:10.449099+0000","last_clean_scrub_stamp":"2023-10-03T17:40:10.449099+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T20:18:04.958666+0000","scrub_duration":0.043225012,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":21871960083,"num_objects":35990,"num_object_clones":30869,"num_object_copies":71980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35990,"num_whiteouts":0,"num_read":331280,"num_read_kb":1247892,"num_write":335351,"num_write_kb":1341398,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2747547648,"data_stored":14583627814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40086,"ondisk_log_size":40086,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":5372,"num_objects":143,"num_object_clones":0,"num_object_copies":286,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":143,"num_whiteouts":0,"num_read":41321,"num_read_kb":34515,"num_write":1623,"num_write_kb":1323,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":983040,"data_stored":10744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7029,"internal_metadata":0},"log_size":10085,"ondisk_log_size":10085,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":7460,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738552,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2156352,"kb_used_data":1342520,"kb_used_omap":2891,"kb_used_meta":810932,"kb_avail":92215488,"statfs":{"total":96636764160,"available":94428659712,"internally_reserved":0,"allocated":1374740480,"data_stored":7292747235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2960945,"internal_metadata":830394831},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,1,3,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 17:49:01 2023","interfaces":[{"interface":"back","average":{"1min":0.33300000000000002,"5min":0.41299999999999998,"15min":0.44},"min":{"1min":0.19600000000000001,"5min":0.16500000000000001,"15min":0.14299999999999999},"max":{"1min":0.751,"5min":3.9950000000000001,"15min":4.4640000000000004},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.38300000000000001,"15min":0.42999999999999999},"min":{"1min":0.224,"5min":0.17999999999999999,"15min":0.17100000000000001},"max":{"1min":0.73399999999999999,"5min":2.9049999999999998,"15min":4.4180000000000001},"last":0.46200000000000002}]},{"osd":1,"last update":"Tue Oct 3 17:49:01 2023","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.54500000000000004,"15min":0.50900000000000001},"min":{"1min":0.22800000000000001,"5min":0.184,"15min":0.16600000000000001},"max":{"1min":1.9390000000000001,"5min":8.1059999999999999,"15min":8.1059999999999999},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.35699999999999998,"5min":0.46899999999999997,"15min":0.45600000000000002},"min":{"1min":0.22600000000000001,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.69999999999999996,"5min":4.1120000000000001,"15min":4.1120000000000001},"last":0.29399999999999998}]}]},{"osd":1,"up_from":8,"seq":34359738550,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1746776,"kb_used_data":662032,"kb_used_omap":1,"kb_used_meta":1084734,"kb_avail":92625064,"statfs":{"total":96636764160,"available":94848065536,"internally_reserved":0,"allocated":677920768,"data_stored":4053183928,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":1110768070},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,2,0,7,6,3,6,6],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 17:49:17 2023","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.33700000000000002,"15min":0.36499999999999999},"min":{"1min":0.185,"5min":0.17199999999999999,"15min":0.161},"max":{"1min":2.7050000000000001,"5min":2.7050000000000001,"15min":4.1879999999999997},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.27200000000000002,"5min":0.38300000000000001,"15min":0.41199999999999998},"min":{"1min":0.20000000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.42899999999999999,"5min":5.8959999999999999,"15min":5.8959999999999999},"last":0.34599999999999997}]},{"osd":2,"last update":"Tue Oct 3 17:49:17 2023","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.47199999999999998,"15min":0.42899999999999999},"min":{"1min":0.17599999999999999,"5min":0.17599999999999999,"15min":0.159},"max":{"1min":4.5519999999999996,"5min":4.5519999999999996,"15min":4.5519999999999996},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.42099999999999999,"15min":0.40500000000000003},"min":{"1min":0.23300000000000001,"5min":0.16400000000000001,"15min":0.16400000000000001},"max":{"1min":2.665,"5min":2.6970000000000001,"15min":3.0699999999999998},"last":0.72999999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738550,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1764404,"kb_used_data":681004,"kb_used_omap":1,"kb_used_meta":1083390,"kb_avail":92607436,"statfs":{"total":96636764160,"available":94830014464,"internally_reserved":0,"allocated":697348096,"data_stored":3237295710,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":1109391814},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,0,2,2,5,4,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 17:49:02 2023","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.39700000000000002,"15min":0.41499999999999998},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.154},"max":{"1min":1.579,"5min":3.4649999999999999,"15min":5.7649999999999997},"last":0.17999999999999999},{"interface":"front","average":{"1min":0.502,"5min":0.42499999999999999,"15min":0.46300000000000002},"min":{"1min":0.23599999999999999,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":1.679,"5min":3.4279999999999999,"15min":9.4770000000000003},"last":0.379}]},{"osd":2,"last update":"Tue Oct 3 17:49:02 2023","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.49399999999999999,"15min":0.51400000000000001},"min":{"1min":0.223,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":1.244,"5min":4.1619999999999999,"15min":9.4149999999999991},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.56799999999999995,"15min":0.495},"min":{"1min":0.224,"5min":0.19700000000000001,"15min":0.189},"max":{"1min":0.66900000000000004,"5min":5.218,"15min":5.218},"last":0.441}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":323584,"data_stored":3509,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":5372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":167936,"data_stored":1863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7029,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":696643584,"data_stored":3237625856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":676966400,"data_stored":4053524499,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1373937664,"data_stored":7292477459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-03T17:49:30.959 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-03T17:34:47.281095+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=34, tm_sec=47, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=40, tm_sec=6, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T17:49:30.959 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T17:49:36.971 INFO:tasks.ceph.osd.0.smithi111.stderr:2023-10-03T17:49:36.972+0000 7ffbac289640 -1 reset not still connected to 0x5563a7ff68f0 2023-10-03T17:49:36.972 INFO:tasks.ceph.osd.0.smithi111.stderr:2023-10-03T17:49:36.972+0000 7ffbac289640 -1 reset not still connected to 0x5563a838e1a0 2023-10-03T17:49:36.972 INFO:tasks.ceph.osd.0.smithi111.stderr:2023-10-03T17:49:36.972+0000 7ffbac289640 -1 reset not still connected to 0x5563a851aea0 2023-10-03T17:49:50.960 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T17:49:51.480 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-03T17:49:51.481 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-10-03T17:49:51.513 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":534,"stamp":"2023-10-03T17:49:50.170905+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":21902972719,"num_objects":35948,"num_object_clones":30682,"num_object_copies":71896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35948,"num_whiteouts":0,"num_read":378893,"num_read_kb":1306950,"num_write":343306,"num_write_kb":1369395,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":50516,"ondisk_log_size":50516,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":5402600,"kb_used_data":2734672,"kb_used_omap":2894,"kb_used_meta":2665009,"kb_avail":277712920,"statfs":{"total":289910292480,"available":284378030080,"internally_reserved":0,"allocated":2800304128,"data_stored":14613619193,"data_compressed":7460,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":2964133,"internal_metadata":2728969563},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,4,3,4,10,10,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":15876096,"num_objects":-223,"num_object_clones":-223,"num_object_copies":-446,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-223,"num_whiteouts":0,"num_read":3217,"num_read_kb":12716,"num_write":3269,"num_write_kb":13076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008095"},"pg_stats":[{"pgid":"2.7","version":"79'1845","reported_seq":3833,"reported_epoch":79,"state":"active+clean","last_fresh":"2023-10-03T17:49:47.446593+0000","last_change":"2023-10-03T17:40:17.407011+0000","last_active":"2023-10-03T17:49:47.446593+0000","last_peered":"2023-10-03T17:49:47.446593+0000","last_clean":"2023-10-03T17:49:47.446593+0000","last_became_active":"2023-10-03T17:34:42.276624+0000","last_became_peered":"2023-10-03T17:34:42.276624+0000","last_unstale":"2023-10-03T17:49:47.446593+0000","last_undegraded":"2023-10-03T17:49:47.446593+0000","last_fullsized":"2023-10-03T17:49:47.446593+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'673","last_scrub_stamp":"2023-10-03T17:40:17.406813+0000","last_deep_scrub":"46'673","last_deep_scrub_stamp":"2023-10-03T17:40:17.406813+0000","last_clean_scrub_stamp":"2023-10-03T17:40:17.406813+0000","objects_scrubbed":7,"log_size":1845,"log_dups_size":0,"ondisk_log_size":1845,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T21:38:35.056889+0000","scrub_duration":0.013778076,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":639,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":5782,"num_read_kb":4774,"num_write":229,"num_write_kb":183,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":903,"num_omap_keys":30,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"79'3066","reported_seq":7670,"reported_epoch":79,"state":"active+clean","last_fresh":"2023-10-03T17:49:47.447807+0000","last_change":"2023-10-03T17:40:13.376519+0000","last_active":"2023-10-03T17:49:47.447807+0000","last_peered":"2023-10-03T17:49:47.447807+0000","last_clean":"2023-10-03T17:49:47.447807+0000","last_became_active":"2023-10-03T17:34:42.271201+0000","last_became_peered":"2023-10-03T17:34:42.271201+0000","last_unstale":"2023-10-03T17:49:47.447807+0000","last_undegraded":"2023-10-03T17:49:47.447807+0000","last_fullsized":"2023-10-03T17:49:47.447807+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'1129","last_scrub_stamp":"2023-10-03T17:40:13.376332+0000","last_deep_scrub":"46'1129","last_deep_scrub_stamp":"2023-10-03T17:40:13.376332+0000","last_clean_scrub_stamp":"2023-10-03T17:40:13.376332+0000","objects_scrubbed":11,"log_size":3066,"log_dups_size":0,"ondisk_log_size":3066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T18:59:17.088585+0000","scrub_duration":0.015998675,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":673,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":11530,"num_read_kb":9857,"num_write":396,"num_write_kb":329,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2912,"num_omap_keys":93,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"78'47","reported_seq":168,"reported_epoch":79,"state":"active+clean","last_fresh":"2023-10-03T17:49:04.392628+0000","last_change":"2023-10-03T17:40:25.496010+0000","last_active":"2023-10-03T17:49:04.392628+0000","last_peered":"2023-10-03T17:49:04.392628+0000","last_clean":"2023-10-03T17:49:04.392628+0000","last_became_active":"2023-10-03T17:34:42.276941+0000","last_became_peered":"2023-10-03T17:34:42.276941+0000","last_unstale":"2023-10-03T17:49:04.392628+0000","last_undegraded":"2023-10-03T17:49:04.392628+0000","last_fullsized":"2023-10-03T17:49:04.392628+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'14","last_scrub_stamp":"2023-10-03T17:40:25.495827+0000","last_deep_scrub":"46'14","last_deep_scrub_stamp":"2023-10-03T17:40:25.495827+0000","last_clean_scrub_stamp":"2023-10-03T17:40:25.495827+0000","objects_scrubbed":7,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:05:44.939663+0000","scrub_duration":0.0133078,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1006,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":48,"num_read_kb":21,"num_write":40,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"79'2446","reported_seq":5762,"reported_epoch":79,"state":"active+clean","last_fresh":"2023-10-03T17:49:47.448045+0000","last_change":"2023-10-03T17:40:26.513902+0000","last_active":"2023-10-03T17:49:47.448045+0000","last_peered":"2023-10-03T17:49:47.448045+0000","last_clean":"2023-10-03T17:49:47.448045+0000","last_became_active":"2023-10-03T17:34:42.276459+0000","last_became_peered":"2023-10-03T17:34:42.276459+0000","last_unstale":"2023-10-03T17:49:47.448045+0000","last_undegraded":"2023-10-03T17:49:47.448045+0000","last_fullsized":"2023-10-03T17:49:47.448045+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'964","last_scrub_stamp":"2023-10-03T17:40:26.513729+0000","last_deep_scrub":"46'964","last_deep_scrub_stamp":"2023-10-03T17:40:26.513729+0000","last_clean_scrub_stamp":"2023-10-03T17:40:26.513729+0000","objects_scrubbed":10,"log_size":2446,"log_dups_size":0,"ondisk_log_size":2446,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:41:33.182751+0000","scrub_duration":0.018190280999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":333,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":8469,"num_read_kb":7079,"num_write":286,"num_write_kb":240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1731,"num_omap_keys":80,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"79'109717","reported_seq":183882,"reported_epoch":79,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T17:49:47.440141+0000","last_change":"2023-10-03T17:49:44.338515+0000","last_active":"2023-10-03T17:49:47.440141+0000","last_peered":"2023-10-03T17:49:47.440141+0000","last_clean":"2023-10-03T17:49:47.440141+0000","last_became_active":"2023-10-03T17:34:48.310701+0000","last_became_peered":"2023-10-03T17:34:48.310701+0000","last_unstale":"2023-10-03T17:49:47.440141+0000","last_undegraded":"2023-10-03T17:49:47.440141+0000","last_fullsized":"2023-10-03T17:49:47.440141+0000","mapping_epoch":16,"log_start":"74'99708","ondisk_log_start":"74'99708","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'80695","last_scrub_stamp":"2023-10-03T17:46:13.982571+0000","last_deep_scrub":"66'80695","last_deep_scrub_stamp":"2023-10-03T17:46:13.982571+0000","last_clean_scrub_stamp":"2023-10-03T17:46:13.982571+0000","objects_scrubbed":3410,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":41,"last_scrub_duration":363,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T23:51:11.410881+0000","scrub_duration":362.26574615599998,"objects_trimmed":704,"snaptrim_duration":39.943908067000002,"stat_sum":{"num_bytes":5645041664,"num_objects":9049,"num_object_clones":7729,"num_object_copies":18098,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9049,"num_whiteouts":0,"num_read":86990,"num_read_kb":327856,"num_write":88371,"num_write_kb":353484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"54","length":"1e"},{"start":"7c","length":"8"},{"start":"8a","length":"2"},{"start":"90","length":"a"},{"start":"a4","length":"9"}]},{"pgid":"2.2","version":"79'2522","reported_seq":5169,"reported_epoch":79,"state":"active+clean","last_fresh":"2023-10-03T17:49:43.108278+0000","last_change":"2023-10-03T17:40:08.675119+0000","last_active":"2023-10-03T17:49:43.108278+0000","last_peered":"2023-10-03T17:49:43.108278+0000","last_clean":"2023-10-03T17:49:43.108278+0000","last_became_active":"2023-10-03T17:34:42.270625+0000","last_became_peered":"2023-10-03T17:34:42.270625+0000","last_unstale":"2023-10-03T17:49:43.108278+0000","last_undegraded":"2023-10-03T17:49:43.108278+0000","last_fullsized":"2023-10-03T17:49:43.108278+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'841","last_scrub_stamp":"2023-10-03T17:40:08.674910+0000","last_deep_scrub":"46'841","last_deep_scrub_stamp":"2023-10-03T17:40:08.674910+0000","last_clean_scrub_stamp":"2023-10-03T17:40:08.674910+0000","objects_scrubbed":11,"log_size":2522,"log_dups_size":0,"ondisk_log_size":2522,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T21:05:42.790614+0000","scrub_duration":0.039189555000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":858,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":7731,"num_read_kb":6397,"num_write":317,"num_write_kb":260,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2564,"num_omap_keys":60,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"79'98357","reported_seq":169869,"reported_epoch":79,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T17:49:47.500802+0000","last_change":"2023-10-03T17:43:28.494717+0000","last_active":"2023-10-03T17:49:47.500802+0000","last_peered":"2023-10-03T17:49:47.500802+0000","last_clean":"2023-10-03T17:49:47.500802+0000","last_became_active":"2023-10-03T17:34:48.310555+0000","last_became_peered":"2023-10-03T17:34:48.310555+0000","last_unstale":"2023-10-03T17:49:47.500802+0000","last_undegraded":"2023-10-03T17:49:47.500802+0000","last_fullsized":"2023-10-03T17:49:47.500802+0000","mapping_epoch":16,"log_start":"74'88306","ondisk_log_start":"74'88306","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_clean_scrub_stamp":"2023-10-03T17:34:47.281095+0000","objects_scrubbed":5972,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":100,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 379s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086289699999999995,"stat_sum":{"num_bytes":5312131072,"num_objects":12149,"num_object_clones":10907,"num_object_copies":24298,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12149,"num_whiteouts":0,"num_read":81902,"num_read_kb":307968,"num_write":82372,"num_write_kb":329488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"79'204","reported_seq":2175,"reported_epoch":79,"state":"active+clean","last_fresh":"2023-10-03T17:49:47.611756+0000","last_change":"2023-10-03T17:40:17.641493+0000","last_active":"2023-10-03T17:49:47.611756+0000","last_peered":"2023-10-03T17:49:47.611756+0000","last_clean":"2023-10-03T17:49:47.611756+0000","last_became_active":"2023-10-03T17:34:42.278196+0000","last_became_peered":"2023-10-03T17:34:42.278196+0000","last_unstale":"2023-10-03T17:49:47.611756+0000","last_undegraded":"2023-10-03T17:49:47.611756+0000","last_fullsized":"2023-10-03T17:49:47.611756+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'30","last_scrub_stamp":"2023-10-03T17:40:17.640969+0000","last_deep_scrub":"46'30","last_deep_scrub_stamp":"2023-10-03T17:40:17.640969+0000","last_clean_scrub_stamp":"2023-10-03T17:40:17.640969+0000","objects_scrubbed":6,"log_size":204,"log_dups_size":0,"ondisk_log_size":204,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:04:02.954896+0000","scrub_duration":0.071054565,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":521,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":5754,"num_read_kb":4755,"num_write":210,"num_write_kb":172,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":903,"num_omap_keys":30,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"79'108537","reported_seq":179216,"reported_epoch":79,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T17:49:47.427336+0000","last_change":"2023-10-03T17:49:43.499390+0000","last_active":"2023-10-03T17:49:47.427336+0000","last_peered":"2023-10-03T17:49:47.427336+0000","last_clean":"2023-10-03T17:49:47.427336+0000","last_became_active":"2023-10-03T17:34:48.310912+0000","last_became_peered":"2023-10-03T17:34:48.310912+0000","last_unstale":"2023-10-03T17:49:47.427336+0000","last_undegraded":"2023-10-03T17:49:47.427336+0000","last_fullsized":"2023-10-03T17:49:47.427336+0000","mapping_epoch":16,"log_start":"74'98468","ondisk_log_start":"74'98468","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'58196","last_scrub_stamp":"2023-10-03T17:43:26.193075+0000","last_deep_scrub":"53'58196","last_deep_scrub_stamp":"2023-10-03T17:43:26.193075+0000","last_clean_scrub_stamp":"2023-10-03T17:43:26.193075+0000","objects_scrubbed":2326,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":9,"last_scrub_duration":196,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:00:09.852937+0000","scrub_duration":195.44739730800001,"objects_trimmed":750,"snaptrim_duration":39.101677600000002,"stat_sum":{"num_bytes":5439143936,"num_objects":6747,"num_object_clones":5476,"num_object_copies":13494,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6747,"num_whiteouts":0,"num_read":83949,"num_read_kb":317244,"num_write":84957,"num_write_kb":339828,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"54","length":"1e"},{"start":"7c","length":"8"},{"start":"8a","length":"2"},{"start":"90","length":"a"},{"start":"a4","length":"a"},{"start":"b8","length":"a"},{"start":"cc","length":"a"},{"start":"e0","length":"a"},{"start":"f4","length":"1"}]},{"pgid":"2.0","version":"78'108","reported_seq":857,"reported_epoch":79,"state":"active+clean","last_fresh":"2023-10-03T17:49:42.326581+0000","last_change":"2023-10-03T17:40:10.708344+0000","last_active":"2023-10-03T17:49:42.326581+0000","last_peered":"2023-10-03T17:49:42.326581+0000","last_clean":"2023-10-03T17:49:42.326581+0000","last_became_active":"2023-10-03T17:34:42.272306+0000","last_became_peered":"2023-10-03T17:34:42.272306+0000","last_unstale":"2023-10-03T17:49:42.326581+0000","last_undegraded":"2023-10-03T17:49:42.326581+0000","last_fullsized":"2023-10-03T17:49:42.326581+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'20","last_scrub_stamp":"2023-10-03T17:40:10.708060+0000","last_deep_scrub":"46'20","last_deep_scrub_stamp":"2023-10-03T17:40:10.708060+0000","last_clean_scrub_stamp":"2023-10-03T17:40:10.708060+0000","objects_scrubbed":7,"log_size":108,"log_dups_size":0,"ondisk_log_size":108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:03:09.097252+0000","scrub_duration":0.042279491000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":805,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":1967,"num_read_kb":1610,"num_write":105,"num_write_kb":82,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":301,"num_omap_keys":10,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"79'108851","reported_seq":180574,"reported_epoch":79,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T17:49:47.510975+0000","last_change":"2023-10-03T17:49:45.631282+0000","last_active":"2023-10-03T17:49:47.510975+0000","last_peered":"2023-10-03T17:49:47.510975+0000","last_clean":"2023-10-03T17:49:47.510975+0000","last_became_active":"2023-10-03T17:34:48.310612+0000","last_became_peered":"2023-10-03T17:34:48.310612+0000","last_unstale":"2023-10-03T17:49:47.510975+0000","last_undegraded":"2023-10-03T17:49:47.510975+0000","last_fullsized":"2023-10-03T17:49:47.510975+0000","mapping_epoch":16,"log_start":"74'98845","ondisk_log_start":"74'98845","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'75807","last_scrub_stamp":"2023-10-03T17:45:43.833310+0000","last_deep_scrub":"61'75807","last_deep_scrub_stamp":"2023-10-03T17:45:43.833310+0000","last_clean_scrub_stamp":"2023-10-03T17:45:43.833310+0000","objects_scrubbed":3607,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":25,"last_scrub_duration":315,"scrub_schedule":"queued for deep scrub","scrub_duration":314.26374841099999,"objects_trimmed":817,"snaptrim_duration":41.220831257,"stat_sum":{"num_bytes":5506060307,"num_objects":7858,"num_object_clones":6570,"num_object_copies":15716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7858,"num_whiteouts":0,"num_read":84625,"num_read_kb":319276,"num_write":85846,"num_write_kb":343378,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"54","length":"1e"},{"start":"7c","length":"8"},{"start":"8a","length":"2"},{"start":"90","length":"a"},{"start":"a4","length":"a"},{"start":"b8","length":"a"},{"start":"cc","length":"5"}]},{"pgid":"2.3","version":"77'44","reported_seq":163,"reported_epoch":79,"state":"active+clean","last_fresh":"2023-10-03T17:49:04.392591+0000","last_change":"2023-10-03T17:40:24.472734+0000","last_active":"2023-10-03T17:49:04.392591+0000","last_peered":"2023-10-03T17:49:04.392591+0000","last_clean":"2023-10-03T17:49:04.392591+0000","last_became_active":"2023-10-03T17:34:42.270194+0000","last_became_peered":"2023-10-03T17:34:42.270194+0000","last_unstale":"2023-10-03T17:49:04.392591+0000","last_undegraded":"2023-10-03T17:49:04.392591+0000","last_fullsized":"2023-10-03T17:49:04.392591+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'8","last_scrub_stamp":"2023-10-03T17:40:24.472573+0000","last_deep_scrub":"46'8","last_deep_scrub_stamp":"2023-10-03T17:40:24.472573+0000","last_clean_scrub_stamp":"2023-10-03T17:40:24.472573+0000","objects_scrubbed":4,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T18:28:04.735033+0000","scrub_duration":0.024990637999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":537,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":40,"num_read_kb":22,"num_write":40,"num_write_kb":26,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"46'99","reported_seq":223,"reported_epoch":78,"state":"active+clean","last_fresh":"2023-10-03T17:49:03.376720+0000","last_change":"2023-10-03T17:40:10.452341+0000","last_active":"2023-10-03T17:49:03.376720+0000","last_peered":"2023-10-03T17:49:03.376720+0000","last_clean":"2023-10-03T17:49:03.376720+0000","last_became_active":"2023-10-03T17:34:39.502393+0000","last_became_peered":"2023-10-03T17:34:39.502393+0000","last_unstale":"2023-10-03T17:49:03.376720+0000","last_undegraded":"2023-10-03T17:49:03.376720+0000","last_fullsized":"2023-10-03T17:49:03.376720+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'99","last_scrub_stamp":"2023-10-03T17:40:10.449099+0000","last_deep_scrub":"46'99","last_deep_scrub_stamp":"2023-10-03T17:40:10.449099+0000","last_clean_scrub_stamp":"2023-10-03T17:40:10.449099+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T20:18:04.958666+0000","scrub_duration":0.043225012,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":21902376979,"num_objects":35803,"num_object_clones":30682,"num_object_copies":71606,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35803,"num_whiteouts":0,"num_read":337466,"num_read_kb":1272344,"num_write":341546,"num_write_kb":1366178,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2798452736,"data_stored":14609854502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40135,"ondisk_log_size":40135,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":5372,"num_objects":143,"num_object_clones":0,"num_object_copies":286,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":143,"num_whiteouts":0,"num_read":41321,"num_read_kb":34515,"num_write":1623,"num_write_kb":1323,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":983040,"data_stored":10744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7029,"internal_metadata":0},"log_size":10282,"ondisk_log_size":10282,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":7460,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738556,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1794840,"kb_used_data":1367248,"kb_used_omap":2891,"kb_used_meta":424692,"kb_avail":92577000,"statfs":{"total":96636764160,"available":94798848000,"internally_reserved":0,"allocated":1400061952,"data_stored":7305154019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2960945,"internal_metadata":434885071},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,0,2,4,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 17:49:01 2023","interfaces":[{"interface":"back","average":{"1min":0.33300000000000002,"5min":0.41299999999999998,"15min":0.44},"min":{"1min":0.19600000000000001,"5min":0.16500000000000001,"15min":0.14299999999999999},"max":{"1min":0.751,"5min":3.9950000000000001,"15min":4.4640000000000004},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.38300000000000001,"15min":0.42999999999999999},"min":{"1min":0.224,"5min":0.17999999999999999,"15min":0.17100000000000001},"max":{"1min":0.73399999999999999,"5min":2.9049999999999998,"15min":4.4180000000000001},"last":0.23999999999999999}]},{"osd":1,"last update":"Tue Oct 3 17:49:01 2023","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.54500000000000004,"15min":0.50900000000000001},"min":{"1min":0.22800000000000001,"5min":0.184,"15min":0.16600000000000001},"max":{"1min":1.9390000000000001,"5min":8.1059999999999999,"15min":8.1059999999999999},"last":0.223},{"interface":"front","average":{"1min":0.35699999999999998,"5min":0.46899999999999997,"15min":0.45600000000000002},"min":{"1min":0.22600000000000001,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.69999999999999996,"5min":4.1120000000000001,"15min":4.1120000000000001},"last":0.38300000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738554,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1794392,"kb_used_data":673552,"kb_used_omap":1,"kb_used_meta":1120830,"kb_avail":92577448,"statfs":{"total":96636764160,"available":94799306752,"internally_reserved":0,"allocated":689717248,"data_stored":4090781112,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":1147730374},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,2,2,4,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 17:49:17 2023","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.33700000000000002,"15min":0.36499999999999999},"min":{"1min":0.185,"5min":0.17199999999999999,"15min":0.161},"max":{"1min":2.7050000000000001,"5min":2.7050000000000001,"15min":4.1879999999999997},"last":0.25800000000000001},{"interface":"front","average":{"1min":0.27200000000000002,"5min":0.38300000000000001,"15min":0.41199999999999998},"min":{"1min":0.20000000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.42899999999999999,"5min":5.8959999999999999,"15min":5.8959999999999999},"last":0.221}]},{"osd":2,"last update":"Tue Oct 3 17:49:17 2023","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.47199999999999998,"15min":0.42899999999999999},"min":{"1min":0.17599999999999999,"5min":0.17599999999999999,"15min":0.159},"max":{"1min":4.5519999999999996,"5min":4.5519999999999996,"15min":4.5519999999999996},"last":0.28799999999999998},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.42099999999999999,"15min":0.40500000000000003},"min":{"1min":0.23300000000000001,"5min":0.16400000000000001,"15min":0.16400000000000001},"max":{"1min":2.665,"5min":2.6970000000000001,"15min":3.0699999999999998},"last":0.30299999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738554,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1813368,"kb_used_data":693872,"kb_used_omap":1,"kb_used_meta":1119486,"kb_avail":92558472,"statfs":{"total":96636764160,"available":94779875328,"internally_reserved":0,"allocated":710524928,"data_stored":3217684062,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":1146354118},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,0,4,4,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 17:49:02 2023","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.39700000000000002,"15min":0.41499999999999998},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.154},"max":{"1min":1.579,"5min":3.4649999999999999,"15min":5.7649999999999997},"last":0.248},{"interface":"front","average":{"1min":0.502,"5min":0.42499999999999999,"15min":0.46300000000000002},"min":{"1min":0.23599999999999999,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":1.679,"5min":3.4279999999999999,"15min":9.4770000000000003},"last":0.96199999999999997}]},{"osd":2,"last update":"Tue Oct 3 17:49:02 2023","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.49399999999999999,"15min":0.51400000000000001},"min":{"1min":0.223,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":1.244,"5min":4.1619999999999999,"15min":9.4149999999999991},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.56799999999999995,"15min":0.495},"min":{"1min":0.224,"5min":0.19700000000000001,"15min":0.189},"max":{"1min":0.66900000000000004,"5min":5.218,"15min":5.218},"last":0.27700000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":323584,"data_stored":3509,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":5372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":167936,"data_stored":1863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7029,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":709959680,"data_stored":3214888960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":689016832,"data_stored":4090630163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1399476224,"data_stored":7304335379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-03T17:49:51.515 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-03T17:34:47.281095+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=34, tm_sec=47, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=40, tm_sec=6, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T17:49:51.515 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T17:49:55.878 INFO:tasks.ceph.osd.0.smithi111.stderr:2023-10-03T17:49:55.879+0000 7ffbac289640 -1 reset not still connected to 0x5563a7ff68f0 2023-10-03T17:49:55.878 INFO:tasks.ceph.osd.0.smithi111.stderr:2023-10-03T17:49:55.879+0000 7ffbac289640 -1 reset not still connected to 0x5563a838e1a0 2023-10-03T17:49:55.878 INFO:tasks.ceph.osd.0.smithi111.stderr:2023-10-03T17:49:55.879+0000 7ffbac289640 -1 reset not still connected to 0x5563a851aea0 2023-10-03T17:50:11.517 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T17:50:11.979 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-03T17:50:11.979 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-10-03T17:50:12.013 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":553,"stamp":"2023-10-03T17:50:10.175094+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":21930751791,"num_objects":36142,"num_object_clones":30876,"num_object_copies":72284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36142,"num_whiteouts":0,"num_read":387750,"num_read_kb":1333026,"num_write":349553,"num_write_kb":1393735,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":50873,"ondisk_log_size":50873,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":5558932,"kb_used_data":2782700,"kb_used_omap":2894,"kb_used_meta":2773297,"kb_avail":277556588,"statfs":{"total":289910292480,"available":284217946112,"internally_reserved":0,"allocated":2849484800,"data_stored":14810372662,"data_compressed":7460,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":2964133,"internal_metadata":2839856475},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,3,4,7,19,34,25,3,1,3],"upper_bound":4096},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":6385613,"num_objects":696,"num_object_clones":697,"num_object_copies":1392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":696,"num_whiteouts":0,"num_read":1618,"num_read_kb":5259,"num_write":1324,"num_write_kb":5149,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.778963"},"pg_stats":[{"pgid":"2.7","version":"85'1885","reported_seq":3995,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-03T17:50:05.428698+0000","last_change":"2023-10-03T17:40:17.407011+0000","last_active":"2023-10-03T17:50:05.428698+0000","last_peered":"2023-10-03T17:50:05.428698+0000","last_clean":"2023-10-03T17:50:05.428698+0000","last_became_active":"2023-10-03T17:34:42.276624+0000","last_became_peered":"2023-10-03T17:34:42.276624+0000","last_unstale":"2023-10-03T17:50:05.428698+0000","last_undegraded":"2023-10-03T17:50:05.428698+0000","last_fullsized":"2023-10-03T17:50:05.428698+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'673","last_scrub_stamp":"2023-10-03T17:40:17.406813+0000","last_deep_scrub":"46'673","last_deep_scrub_stamp":"2023-10-03T17:40:17.406813+0000","last_clean_scrub_stamp":"2023-10-03T17:40:17.406813+0000","objects_scrubbed":7,"log_size":1885,"log_dups_size":0,"ondisk_log_size":1885,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T21:38:35.056889+0000","scrub_duration":0.013778076,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":687,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":6213,"num_read_kb":5133,"num_write":261,"num_write_kb":207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":903,"num_omap_keys":30,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"83'3128","reported_seq":8004,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-03T17:50:05.401571+0000","last_change":"2023-10-03T17:40:13.376519+0000","last_active":"2023-10-03T17:50:05.401571+0000","last_peered":"2023-10-03T17:50:05.401571+0000","last_clean":"2023-10-03T17:50:05.401571+0000","last_became_active":"2023-10-03T17:34:42.271201+0000","last_became_peered":"2023-10-03T17:34:42.271201+0000","last_unstale":"2023-10-03T17:50:05.401571+0000","last_undegraded":"2023-10-03T17:50:05.401571+0000","last_fullsized":"2023-10-03T17:50:05.401571+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'1129","last_scrub_stamp":"2023-10-03T17:40:13.376332+0000","last_deep_scrub":"46'1129","last_deep_scrub_stamp":"2023-10-03T17:40:13.376332+0000","last_clean_scrub_stamp":"2023-10-03T17:40:13.376332+0000","objects_scrubbed":11,"log_size":3128,"log_dups_size":0,"ondisk_log_size":3128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T18:59:17.088585+0000","scrub_duration":0.015998675,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":723,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":12327,"num_read_kb":10534,"num_write":446,"num_write_kb":367,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2912,"num_omap_keys":93,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"85'50","reported_seq":180,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-03T17:50:05.711141+0000","last_change":"2023-10-03T17:40:25.496010+0000","last_active":"2023-10-03T17:50:05.711141+0000","last_peered":"2023-10-03T17:50:05.711141+0000","last_clean":"2023-10-03T17:50:05.711141+0000","last_became_active":"2023-10-03T17:34:42.276941+0000","last_became_peered":"2023-10-03T17:34:42.276941+0000","last_unstale":"2023-10-03T17:50:05.711141+0000","last_undegraded":"2023-10-03T17:50:05.711141+0000","last_fullsized":"2023-10-03T17:50:05.711141+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'14","last_scrub_stamp":"2023-10-03T17:40:25.495827+0000","last_deep_scrub":"46'14","last_deep_scrub_stamp":"2023-10-03T17:40:25.495827+0000","last_clean_scrub_stamp":"2023-10-03T17:40:25.495827+0000","objects_scrubbed":7,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:05:44.939663+0000","scrub_duration":0.0133078,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":957,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":50,"num_read_kb":23,"num_write":43,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"85'2500","reported_seq":6017,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-03T17:50:05.428688+0000","last_change":"2023-10-03T17:40:26.513902+0000","last_active":"2023-10-03T17:50:05.428688+0000","last_peered":"2023-10-03T17:50:05.428688+0000","last_clean":"2023-10-03T17:50:05.428688+0000","last_became_active":"2023-10-03T17:34:42.276459+0000","last_became_peered":"2023-10-03T17:34:42.276459+0000","last_unstale":"2023-10-03T17:50:05.428688+0000","last_undegraded":"2023-10-03T17:50:05.428688+0000","last_fullsized":"2023-10-03T17:50:05.428688+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'964","last_scrub_stamp":"2023-10-03T17:40:26.513729+0000","last_deep_scrub":"46'964","last_deep_scrub_stamp":"2023-10-03T17:40:26.513729+0000","last_clean_scrub_stamp":"2023-10-03T17:40:26.513729+0000","objects_scrubbed":10,"log_size":2500,"log_dups_size":0,"ondisk_log_size":2500,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:41:33.182751+0000","scrub_duration":0.018190280999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":537,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":9081,"num_read_kb":7595,"num_write":330,"num_write_kb":274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1731,"num_omap_keys":80,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"86'112486","reported_seq":187686,"reported_epoch":86,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T17:50:07.667418+0000","last_change":"2023-10-03T17:50:06.902200+0000","last_active":"2023-10-03T17:50:07.667418+0000","last_peered":"2023-10-03T17:50:07.667418+0000","last_clean":"2023-10-03T17:50:07.667418+0000","last_became_active":"2023-10-03T17:34:48.310701+0000","last_became_peered":"2023-10-03T17:34:48.310701+0000","last_unstale":"2023-10-03T17:50:07.667418+0000","last_undegraded":"2023-10-03T17:50:07.667418+0000","last_fullsized":"2023-10-03T17:50:07.667418+0000","mapping_epoch":16,"log_start":"74'102424","ondisk_log_start":"74'102424","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'80695","last_scrub_stamp":"2023-10-03T17:46:13.982571+0000","last_deep_scrub":"66'80695","last_deep_scrub_stamp":"2023-10-03T17:46:13.982571+0000","last_clean_scrub_stamp":"2023-10-03T17:46:13.982571+0000","objects_scrubbed":3410,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":52,"last_scrub_duration":363,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:45:08.818908+0000","scrub_duration":362.26574615599998,"objects_trimmed":22,"snaptrim_duration":0.48427703799999999,"stat_sum":{"num_bytes":5652078592,"num_objects":9005,"num_object_clones":7685,"num_object_copies":18010,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9005,"num_whiteouts":0,"num_read":88567,"num_read_kb":334052,"num_write":89933,"num_write_kb":359732,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"54","length":"1e"},{"start":"7c","length":"8"},{"start":"8a","length":"2"},{"start":"90","length":"a"},{"start":"a4","length":"a"},{"start":"b8","length":"6"}]},{"pgid":"2.2","version":"86'2597","reported_seq":5404,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-03T17:50:07.625068+0000","last_change":"2023-10-03T17:40:08.675119+0000","last_active":"2023-10-03T17:50:07.625068+0000","last_peered":"2023-10-03T17:50:07.625068+0000","last_clean":"2023-10-03T17:50:07.625068+0000","last_became_active":"2023-10-03T17:34:42.270625+0000","last_became_peered":"2023-10-03T17:34:42.270625+0000","last_unstale":"2023-10-03T17:50:07.625068+0000","last_undegraded":"2023-10-03T17:50:07.625068+0000","last_fullsized":"2023-10-03T17:50:07.625068+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'841","last_scrub_stamp":"2023-10-03T17:40:08.674910+0000","last_deep_scrub":"46'841","last_deep_scrub_stamp":"2023-10-03T17:40:08.674910+0000","last_clean_scrub_stamp":"2023-10-03T17:40:08.674910+0000","objects_scrubbed":11,"log_size":2597,"log_dups_size":0,"ondisk_log_size":2597,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T21:05:42.790614+0000","scrub_duration":0.039189555000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":909,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":8306,"num_read_kb":6876,"num_write":360,"num_write_kb":292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2564,"num_omap_keys":60,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"86'100427","reported_seq":173150,"reported_epoch":86,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T17:50:07.730243+0000","last_change":"2023-10-03T17:43:28.494717+0000","last_active":"2023-10-03T17:50:07.730243+0000","last_peered":"2023-10-03T17:50:07.730243+0000","last_clean":"2023-10-03T17:50:07.730243+0000","last_became_active":"2023-10-03T17:34:48.310555+0000","last_became_peered":"2023-10-03T17:34:48.310555+0000","last_unstale":"2023-10-03T17:50:07.730243+0000","last_undegraded":"2023-10-03T17:50:07.730243+0000","last_fullsized":"2023-10-03T17:50:07.730243+0000","mapping_epoch":16,"log_start":"74'90409","ondisk_log_start":"74'90409","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_clean_scrub_stamp":"2023-10-03T17:34:47.281095+0000","objects_scrubbed":6339,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":118,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 399s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086289699999999995,"stat_sum":{"num_bytes":5318397952,"num_objects":12465,"num_object_clones":11223,"num_object_copies":24930,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12465,"num_whiteouts":0,"num_read":83396,"num_read_kb":313840,"num_write":83795,"num_write_kb":335180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"84'215","reported_seq":2261,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-03T17:50:05.460856+0000","last_change":"2023-10-03T17:40:17.641493+0000","last_active":"2023-10-03T17:50:05.460856+0000","last_peered":"2023-10-03T17:50:05.460856+0000","last_clean":"2023-10-03T17:50:05.460856+0000","last_became_active":"2023-10-03T17:34:42.278196+0000","last_became_peered":"2023-10-03T17:34:42.278196+0000","last_unstale":"2023-10-03T17:50:05.460856+0000","last_undegraded":"2023-10-03T17:50:05.460856+0000","last_fullsized":"2023-10-03T17:50:05.460856+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'30","last_scrub_stamp":"2023-10-03T17:40:17.640969+0000","last_deep_scrub":"46'30","last_deep_scrub_stamp":"2023-10-03T17:40:17.640969+0000","last_clean_scrub_stamp":"2023-10-03T17:40:17.640969+0000","objects_scrubbed":6,"log_size":215,"log_dups_size":0,"ondisk_log_size":215,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:04:02.954896+0000","scrub_duration":0.071054565,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":470,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":6017,"num_read_kb":4970,"num_write":222,"num_write_kb":180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":903,"num_omap_keys":30,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"86'111258","reported_seq":182849,"reported_epoch":86,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T17:50:07.657955+0000","last_change":"2023-10-03T17:50:06.243661+0000","last_active":"2023-10-03T17:50:07.657955+0000","last_peered":"2023-10-03T17:50:07.657955+0000","last_clean":"2023-10-03T17:50:07.657955+0000","last_became_active":"2023-10-03T17:34:48.310912+0000","last_became_peered":"2023-10-03T17:34:48.310912+0000","last_unstale":"2023-10-03T17:50:07.657955+0000","last_undegraded":"2023-10-03T17:50:07.657955+0000","last_fullsized":"2023-10-03T17:50:07.657955+0000","mapping_epoch":16,"log_start":"74'101176","ondisk_log_start":"74'101176","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'58196","last_scrub_stamp":"2023-10-03T17:43:26.193075+0000","last_deep_scrub":"53'58196","last_deep_scrub_stamp":"2023-10-03T17:43:26.193075+0000","last_clean_scrub_stamp":"2023-10-03T17:43:26.193075+0000","objects_scrubbed":2326,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":20,"last_scrub_duration":196,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T21:06:25.996902+0000","scrub_duration":195.44739730800001,"objects_trimmed":12,"snaptrim_duration":0.82816528,"stat_sum":{"num_bytes":5446340608,"num_objects":6708,"num_object_clones":5437,"num_object_copies":13416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6708,"num_whiteouts":0,"num_read":85394,"num_read_kb":322936,"num_write":86490,"num_write_kb":345960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"54","length":"1e"},{"start":"7c","length":"8"},{"start":"8a","length":"2"},{"start":"90","length":"a"},{"start":"a4","length":"a"},{"start":"b8","length":"a"},{"start":"cc","length":"a"},{"start":"e0","length":"a"},{"start":"f4","length":"8"}]},{"pgid":"2.0","version":"86'116","reported_seq":911,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-03T17:50:06.820811+0000","last_change":"2023-10-03T17:40:10.708344+0000","last_active":"2023-10-03T17:50:06.820811+0000","last_peered":"2023-10-03T17:50:06.820811+0000","last_clean":"2023-10-03T17:50:06.820811+0000","last_became_active":"2023-10-03T17:34:42.272306+0000","last_became_peered":"2023-10-03T17:34:42.272306+0000","last_unstale":"2023-10-03T17:50:06.820811+0000","last_undegraded":"2023-10-03T17:50:06.820811+0000","last_fullsized":"2023-10-03T17:50:06.820811+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'20","last_scrub_stamp":"2023-10-03T17:40:10.708060+0000","last_deep_scrub":"46'20","last_deep_scrub_stamp":"2023-10-03T17:40:10.708060+0000","last_clean_scrub_stamp":"2023-10-03T17:40:10.708060+0000","objects_scrubbed":7,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:03:09.097252+0000","scrub_duration":0.042279491000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":704,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":2112,"num_read_kb":1731,"num_write":116,"num_write_kb":89,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":301,"num_omap_keys":10,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"86'111531","reported_seq":184247,"reported_epoch":86,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T17:50:07.721252+0000","last_change":"2023-10-03T17:50:04.798706+0000","last_active":"2023-10-03T17:50:07.721252+0000","last_peered":"2023-10-03T17:50:07.721252+0000","last_clean":"2023-10-03T17:50:07.721252+0000","last_became_active":"2023-10-03T17:34:48.310612+0000","last_became_peered":"2023-10-03T17:34:48.310612+0000","last_unstale":"2023-10-03T17:50:07.721252+0000","last_undegraded":"2023-10-03T17:50:07.721252+0000","last_fullsized":"2023-10-03T17:50:07.721252+0000","mapping_epoch":16,"log_start":"74'101457","ondisk_log_start":"74'101457","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'75807","last_scrub_stamp":"2023-10-03T17:45:43.833310+0000","last_deep_scrub":"61'75807","last_deep_scrub_stamp":"2023-10-03T17:45:43.833310+0000","last_clean_scrub_stamp":"2023-10-03T17:45:43.833310+0000","objects_scrubbed":3607,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":37,"last_scrub_duration":315,"scrub_schedule":"queued for deep scrub","scrub_duration":314.26374841099999,"objects_trimmed":26,"snaptrim_duration":0.39291596000000001,"stat_sum":{"num_bytes":5513338899,"num_objects":7819,"num_object_clones":6531,"num_object_copies":15638,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7819,"num_whiteouts":0,"num_read":86138,"num_read_kb":325220,"num_write":87377,"num_write_kb":349502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"54","length":"1e"},{"start":"7c","length":"8"},{"start":"8a","length":"2"},{"start":"90","length":"a"},{"start":"a4","length":"a"},{"start":"b8","length":"a"},{"start":"cc","length":"a"},{"start":"e0","length":"1"}]},{"pgid":"2.3","version":"82'47","reported_seq":176,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-03T17:50:05.402364+0000","last_change":"2023-10-03T17:40:24.472734+0000","last_active":"2023-10-03T17:50:05.402364+0000","last_peered":"2023-10-03T17:50:05.402364+0000","last_clean":"2023-10-03T17:50:05.402364+0000","last_became_active":"2023-10-03T17:34:42.270194+0000","last_became_peered":"2023-10-03T17:34:42.270194+0000","last_unstale":"2023-10-03T17:50:05.402364+0000","last_undegraded":"2023-10-03T17:50:05.402364+0000","last_fullsized":"2023-10-03T17:50:05.402364+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'8","last_scrub_stamp":"2023-10-03T17:40:24.472573+0000","last_deep_scrub":"46'8","last_deep_scrub_stamp":"2023-10-03T17:40:24.472573+0000","last_clean_scrub_stamp":"2023-10-03T17:40:24.472573+0000","objects_scrubbed":4,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T18:28:04.735033+0000","scrub_duration":0.024990637999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":385,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":43,"num_read_kb":25,"num_write":43,"num_write_kb":26,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"46'99","reported_seq":230,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-03T17:50:05.417584+0000","last_change":"2023-10-03T17:40:10.452341+0000","last_active":"2023-10-03T17:50:05.417584+0000","last_peered":"2023-10-03T17:50:05.417584+0000","last_clean":"2023-10-03T17:50:05.417584+0000","last_became_active":"2023-10-03T17:34:39.502393+0000","last_became_peered":"2023-10-03T17:34:39.502393+0000","last_unstale":"2023-10-03T17:50:05.417584+0000","last_undegraded":"2023-10-03T17:50:05.417584+0000","last_fullsized":"2023-10-03T17:50:05.417584+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'99","last_scrub_stamp":"2023-10-03T17:40:10.449099+0000","last_deep_scrub":"46'99","last_deep_scrub_stamp":"2023-10-03T17:40:10.449099+0000","last_clean_scrub_stamp":"2023-10-03T17:40:10.449099+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T20:18:04.958666+0000","scrub_duration":0.043225012,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":21930156051,"num_objects":35997,"num_object_clones":30876,"num_object_copies":71994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35997,"num_whiteouts":0,"num_read":343495,"num_read_kb":1296048,"num_write":347595,"num_write_kb":1390374,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2847817728,"data_stored":14869958694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40236,"ondisk_log_size":40236,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":5372,"num_objects":143,"num_object_clones":0,"num_object_copies":286,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":143,"num_whiteouts":0,"num_read":44149,"num_read_kb":36887,"num_write":1821,"num_write_kb":1467,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":983040,"data_stored":10744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7029,"internal_metadata":0},"log_size":10538,"ondisk_log_size":10538,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":7460,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738560,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1894928,"kb_used_data":1390972,"kb_used_omap":2891,"kb_used_meta":501044,"kb_avail":92476912,"statfs":{"total":96636764160,"available":94696357888,"internally_reserved":0,"allocated":1424355328,"data_stored":7418412914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2960945,"internal_metadata":513069519},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,1,3,6,8,8,3,1,3],"upper_bound":4096},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 17:50:02 2023","interfaces":[{"interface":"back","average":{"1min":1.2230000000000001,"5min":0.60599999999999998,"15min":0.48699999999999999},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.14299999999999999},"max":{"1min":14.787000000000001,"5min":14.787000000000001,"15min":14.787000000000001},"last":0.26900000000000002},{"interface":"front","average":{"1min":1.4319999999999999,"5min":0.60899999999999999,"15min":0.48899999999999999},"min":{"1min":0.185,"5min":0.17999999999999999,"15min":0.17100000000000001},"max":{"1min":14.919,"5min":14.919,"15min":14.919},"last":0.40600000000000003}]},{"osd":1,"last update":"Tue Oct 3 17:50:02 2023","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.52400000000000002,"15min":0.51400000000000001},"min":{"1min":0.221,"5min":0.184,"15min":0.16600000000000001},"max":{"1min":6.7539999999999996,"5min":6.7539999999999996,"15min":8.1059999999999999},"last":0.23400000000000001},{"interface":"front","average":{"1min":0.89800000000000002,"5min":0.57899999999999996,"15min":0.47599999999999998},"min":{"1min":0.22700000000000001,"5min":0.223,"15min":0.17699999999999999},"max":{"1min":5.194,"5min":5.194,"15min":5.194},"last":0.28000000000000003}]}]},{"osd":1,"up_from":8,"seq":34359738558,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1820240,"kb_used_data":685508,"kb_used_omap":1,"kb_used_meta":1134718,"kb_avail":92551600,"statfs":{"total":96636764160,"available":94772838400,"internally_reserved":0,"allocated":701960192,"data_stored":4179497079,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":1161951686},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,2,9,15,15],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 17:49:17 2023","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.33700000000000002,"15min":0.36499999999999999},"min":{"1min":0.185,"5min":0.17199999999999999,"15min":0.161},"max":{"1min":2.7050000000000001,"5min":2.7050000000000001,"15min":4.1879999999999997},"last":0.22500000000000001},{"interface":"front","average":{"1min":0.27200000000000002,"5min":0.38300000000000001,"15min":0.41199999999999998},"min":{"1min":0.20000000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.42899999999999999,"5min":5.8959999999999999,"15min":5.8959999999999999},"last":0.254}]},{"osd":2,"last update":"Tue Oct 3 17:49:17 2023","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.47199999999999998,"15min":0.42899999999999999},"min":{"1min":0.17599999999999999,"5min":0.17599999999999999,"15min":0.159},"max":{"1min":4.5519999999999996,"5min":4.5519999999999996,"15min":4.5519999999999996},"last":0.29299999999999998},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.42099999999999999,"15min":0.40500000000000003},"min":{"1min":0.23300000000000001,"5min":0.16400000000000001,"15min":0.16400000000000001},"max":{"1min":2.665,"5min":2.6970000000000001,"15min":3.0699999999999998},"last":0.27500000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738558,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1843764,"kb_used_data":706220,"kb_used_omap":1,"kb_used_meta":1137534,"kb_avail":92528076,"statfs":{"total":96636764160,"available":94748749824,"internally_reserved":0,"allocated":723169280,"data_stored":3212462669,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":1164835270},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,0,2,4,11,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 17:50:03 2023","interfaces":[{"interface":"back","average":{"1min":0.29499999999999998,"5min":0.38100000000000001,"15min":0.40500000000000003},"min":{"1min":0.17999999999999999,"5min":0.17499999999999999,"15min":0.154},"max":{"1min":0.48999999999999999,"5min":3.4649999999999999,"15min":5.7649999999999997},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.46999999999999997,"15min":0.46700000000000003},"min":{"1min":0.23999999999999999,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":3.1320000000000001,"5min":3.4279999999999999,"15min":9.4770000000000003},"last":0.63300000000000001}]},{"osd":2,"last update":"Tue Oct 3 17:50:03 2023","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.54600000000000004,"15min":0.51400000000000001},"min":{"1min":0.25,"5min":0.20000000000000001,"15min":0.19900000000000001},"max":{"1min":3.2149999999999999,"5min":4.1619999999999999,"15min":9.4149999999999991},"last":0.23300000000000001},{"interface":"front","average":{"1min":0.38,"5min":0.55700000000000005,"15min":0.48399999999999999},"min":{"1min":0.23899999999999999,"5min":0.224,"15min":0.189},"max":{"1min":0.73699999999999999,"5min":5.218,"15min":5.218},"last":0.60099999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":348160,"data_stored":3813,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":5372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":143360,"data_stored":1559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7029,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":722640896,"data_stored":3234123776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":701136896,"data_stored":4195065875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1424039936,"data_stored":7440769043,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-03T17:50:12.015 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-03T17:34:47.281095+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=34, tm_sec=47, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=40, tm_sec=6, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T17:50:12.015 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T17:50:30.007 INFO:tasks.ceph.mgr.x.smithi111.stderr:2023-10-03T17:50:30.007+0000 7fe48bb97640 -1 calc_pg_upmaps abort due to max <= 0 2023-10-03T17:50:32.016 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T17:50:32.599 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-03T17:50:32.600 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-10-03T17:50:32.630 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":564,"stamp":"2023-10-03T17:50:32.181341+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":21955774255,"num_objects":37763,"num_object_clones":32497,"num_object_copies":75526,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37763,"num_whiteouts":0,"num_read":394486,"num_read_kb":1354489,"num_write":355084,"num_write_kb":1415787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":50929,"ondisk_log_size":50929,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":5801676,"kb_used_data":2826996,"kb_used_omap":2894,"kb_used_meta":2971761,"kb_avail":277313844,"statfs":{"total":289910292480,"available":283969376256,"internally_reserved":0,"allocated":2894843904,"data_stored":15844551900,"data_compressed":7460,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":2964133,"internal_metadata":3043083611},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,9,2,10,18,15,15,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":13189120,"num_objects":505,"num_object_clones":505,"num_object_copies":1010,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":505,"num_whiteouts":0,"num_read":2769,"num_read_kb":10564,"num_write":2917,"num_write_kb":11668,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001956"},"pg_stats":[{"pgid":"2.7","version":"88'1914","reported_seq":4071,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-03T17:50:27.205299+0000","last_change":"2023-10-03T17:40:17.407011+0000","last_active":"2023-10-03T17:50:27.205299+0000","last_peered":"2023-10-03T17:50:27.205299+0000","last_clean":"2023-10-03T17:50:27.205299+0000","last_became_active":"2023-10-03T17:34:42.276624+0000","last_became_peered":"2023-10-03T17:34:42.276624+0000","last_unstale":"2023-10-03T17:50:27.205299+0000","last_undegraded":"2023-10-03T17:50:27.205299+0000","last_fullsized":"2023-10-03T17:50:27.205299+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'673","last_scrub_stamp":"2023-10-03T17:40:17.406813+0000","last_deep_scrub":"46'673","last_deep_scrub_stamp":"2023-10-03T17:40:17.406813+0000","last_clean_scrub_stamp":"2023-10-03T17:40:17.406813+0000","objects_scrubbed":7,"log_size":1914,"log_dups_size":0,"ondisk_log_size":1914,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T21:38:35.056889+0000","scrub_duration":0.013778076,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":789,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":6411,"num_read_kb":5295,"num_write":263,"num_write_kb":209,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":903,"num_omap_keys":30,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"88'3173","reported_seq":8168,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-03T17:50:27.214156+0000","last_change":"2023-10-03T17:40:13.376519+0000","last_active":"2023-10-03T17:50:27.214156+0000","last_peered":"2023-10-03T17:50:27.214156+0000","last_clean":"2023-10-03T17:50:27.214156+0000","last_became_active":"2023-10-03T17:34:42.271201+0000","last_became_peered":"2023-10-03T17:34:42.271201+0000","last_unstale":"2023-10-03T17:50:27.214156+0000","last_undegraded":"2023-10-03T17:50:27.214156+0000","last_fullsized":"2023-10-03T17:50:27.214156+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'1129","last_scrub_stamp":"2023-10-03T17:40:13.376332+0000","last_deep_scrub":"46'1129","last_deep_scrub_stamp":"2023-10-03T17:40:13.376332+0000","last_clean_scrub_stamp":"2023-10-03T17:40:13.376332+0000","objects_scrubbed":11,"log_size":3173,"log_dups_size":0,"ondisk_log_size":3173,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T18:59:17.088585+0000","scrub_duration":0.015998675,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":723,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":12741,"num_read_kb":10887,"num_write":446,"num_write_kb":367,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2912,"num_omap_keys":93,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"85'50","reported_seq":182,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-03T17:50:08.417281+0000","last_change":"2023-10-03T17:40:25.496010+0000","last_active":"2023-10-03T17:50:08.417281+0000","last_peered":"2023-10-03T17:50:08.417281+0000","last_clean":"2023-10-03T17:50:08.417281+0000","last_became_active":"2023-10-03T17:34:42.276941+0000","last_became_peered":"2023-10-03T17:34:42.276941+0000","last_unstale":"2023-10-03T17:50:08.417281+0000","last_undegraded":"2023-10-03T17:50:08.417281+0000","last_fullsized":"2023-10-03T17:50:08.417281+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'14","last_scrub_stamp":"2023-10-03T17:40:25.495827+0000","last_deep_scrub":"46'14","last_deep_scrub_stamp":"2023-10-03T17:40:25.495827+0000","last_clean_scrub_stamp":"2023-10-03T17:40:25.495827+0000","objects_scrubbed":7,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:05:44.939663+0000","scrub_duration":0.0133078,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":957,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":50,"num_read_kb":23,"num_write":43,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"88'2536","reported_seq":6139,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-03T17:50:27.212914+0000","last_change":"2023-10-03T17:40:26.513902+0000","last_active":"2023-10-03T17:50:27.212914+0000","last_peered":"2023-10-03T17:50:27.212914+0000","last_clean":"2023-10-03T17:50:27.212914+0000","last_became_active":"2023-10-03T17:34:42.276459+0000","last_became_peered":"2023-10-03T17:34:42.276459+0000","last_unstale":"2023-10-03T17:50:27.212914+0000","last_undegraded":"2023-10-03T17:50:27.212914+0000","last_fullsized":"2023-10-03T17:50:27.212914+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'964","last_scrub_stamp":"2023-10-03T17:40:26.513729+0000","last_deep_scrub":"46'964","last_deep_scrub_stamp":"2023-10-03T17:40:26.513729+0000","last_clean_scrub_stamp":"2023-10-03T17:40:26.513729+0000","objects_scrubbed":10,"log_size":2536,"log_dups_size":0,"ondisk_log_size":2536,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:41:33.182751+0000","scrub_duration":0.018190280999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":537,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":9369,"num_read_kb":7834,"num_write":330,"num_write_kb":274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1731,"num_omap_keys":80,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"88'115152","reported_seq":190892,"reported_epoch":88,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T17:50:27.857341+0000","last_change":"2023-10-03T17:50:25.710781+0000","last_active":"2023-10-03T17:50:27.857341+0000","last_peered":"2023-10-03T17:50:27.857341+0000","last_clean":"2023-10-03T17:50:27.857341+0000","last_became_active":"2023-10-03T17:34:48.310701+0000","last_became_peered":"2023-10-03T17:34:48.310701+0000","last_unstale":"2023-10-03T17:50:27.857341+0000","last_undegraded":"2023-10-03T17:50:27.857341+0000","last_fullsized":"2023-10-03T17:50:27.857341+0000","mapping_epoch":16,"log_start":"79'105138","ondisk_log_start":"79'105138","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'80695","last_scrub_stamp":"2023-10-03T17:46:13.982571+0000","last_deep_scrub":"66'80695","last_deep_scrub_stamp":"2023-10-03T17:46:13.982571+0000","last_clean_scrub_stamp":"2023-10-03T17:46:13.982571+0000","objects_scrubbed":3410,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":50,"last_scrub_duration":363,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:57:32.446767+0000","scrub_duration":362.26574615599998,"objects_trimmed":251,"snaptrim_duration":16.283708624999999,"stat_sum":{"num_bytes":5658902528,"num_objects":9400,"num_object_clones":8080,"num_object_copies":18800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9400,"num_whiteouts":0,"num_read":89884,"num_read_kb":339212,"num_write":91385,"num_write_kb":365540,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"54","length":"1e"},{"start":"7c","length":"8"},{"start":"8a","length":"2"},{"start":"90","length":"a"},{"start":"a4","length":"a"},{"start":"b8","length":"a"}]},{"pgid":"2.2","version":"88'2629","reported_seq":5500,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-03T17:50:24.922555+0000","last_change":"2023-10-03T17:40:08.675119+0000","last_active":"2023-10-03T17:50:24.922555+0000","last_peered":"2023-10-03T17:50:24.922555+0000","last_clean":"2023-10-03T17:50:24.922555+0000","last_became_active":"2023-10-03T17:34:42.270625+0000","last_became_peered":"2023-10-03T17:34:42.270625+0000","last_unstale":"2023-10-03T17:50:24.922555+0000","last_undegraded":"2023-10-03T17:50:24.922555+0000","last_fullsized":"2023-10-03T17:50:24.922555+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'841","last_scrub_stamp":"2023-10-03T17:40:08.674910+0000","last_deep_scrub":"46'841","last_deep_scrub_stamp":"2023-10-03T17:40:08.674910+0000","last_clean_scrub_stamp":"2023-10-03T17:40:08.674910+0000","objects_scrubbed":11,"log_size":2629,"log_dups_size":0,"ondisk_log_size":2629,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T21:05:42.790614+0000","scrub_duration":0.039189555000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":909,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":8572,"num_read_kb":7095,"num_write":360,"num_write_kb":292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2564,"num_omap_keys":60,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"88'102638","reported_seq":176026,"reported_epoch":88,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T17:50:27.913284+0000","last_change":"2023-10-03T17:43:28.494717+0000","last_active":"2023-10-03T17:50:27.913284+0000","last_peered":"2023-10-03T17:50:27.913284+0000","last_clean":"2023-10-03T17:50:27.913284+0000","last_became_active":"2023-10-03T17:34:48.310555+0000","last_became_peered":"2023-10-03T17:34:48.310555+0000","last_unstale":"2023-10-03T17:50:27.913284+0000","last_undegraded":"2023-10-03T17:50:27.913284+0000","last_fullsized":"2023-10-03T17:50:27.913284+0000","mapping_epoch":16,"log_start":"74'92615","ondisk_log_start":"74'92615","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_clean_scrub_stamp":"2023-10-03T17:34:47.281095+0000","objects_scrubbed":6644,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":120,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 419s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086289699999999995,"stat_sum":{"num_bytes":5324353536,"num_objects":13092,"num_object_clones":11850,"num_object_copies":26184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13092,"num_whiteouts":0,"num_read":84670,"num_read_kb":318816,"num_write":85108,"num_write_kb":340432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"88'226","reported_seq":2364,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-03T17:50:23.286175+0000","last_change":"2023-10-03T17:40:17.641493+0000","last_active":"2023-10-03T17:50:23.286175+0000","last_peered":"2023-10-03T17:50:23.286175+0000","last_clean":"2023-10-03T17:50:23.286175+0000","last_became_active":"2023-10-03T17:34:42.278196+0000","last_became_peered":"2023-10-03T17:34:42.278196+0000","last_unstale":"2023-10-03T17:50:23.286175+0000","last_undegraded":"2023-10-03T17:50:23.286175+0000","last_fullsized":"2023-10-03T17:50:23.286175+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'30","last_scrub_stamp":"2023-10-03T17:40:17.640969+0000","last_deep_scrub":"46'30","last_deep_scrub_stamp":"2023-10-03T17:40:17.640969+0000","last_clean_scrub_stamp":"2023-10-03T17:40:17.640969+0000","objects_scrubbed":6,"log_size":226,"log_dups_size":0,"ondisk_log_size":226,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:04:02.954896+0000","scrub_duration":0.071054565,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":419,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":6384,"num_read_kb":5277,"num_write":241,"num_write_kb":194,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":903,"num_omap_keys":30,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"88'113934","reported_seq":186054,"reported_epoch":88,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T17:50:27.863087+0000","last_change":"2023-10-03T17:50:24.053688+0000","last_active":"2023-10-03T17:50:27.863087+0000","last_peered":"2023-10-03T17:50:27.863087+0000","last_clean":"2023-10-03T17:50:27.863087+0000","last_became_active":"2023-10-03T17:34:48.310912+0000","last_became_peered":"2023-10-03T17:34:48.310912+0000","last_unstale":"2023-10-03T17:50:27.863087+0000","last_undegraded":"2023-10-03T17:50:27.863087+0000","last_fullsized":"2023-10-03T17:50:27.863087+0000","mapping_epoch":16,"log_start":"79'103891","ondisk_log_start":"79'103891","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'58196","last_scrub_stamp":"2023-10-03T17:43:26.193075+0000","last_deep_scrub":"53'58196","last_deep_scrub_stamp":"2023-10-03T17:43:26.193075+0000","last_clean_scrub_stamp":"2023-10-03T17:43:26.193075+0000","objects_scrubbed":2326,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":17,"last_scrub_duration":196,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:16:36.461572+0000","scrub_duration":195.44739730800001,"objects_trimmed":317,"snaptrim_duration":14.620919958,"stat_sum":{"num_bytes":5452709888,"num_objects":6990,"num_object_clones":5719,"num_object_copies":13980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6990,"num_whiteouts":0,"num_read":86703,"num_read_kb":328060,"num_write":87843,"num_write_kb":351372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"54","length":"1e"},{"start":"7c","length":"8"},{"start":"8a","length":"2"},{"start":"90","length":"a"},{"start":"a4","length":"a"},{"start":"b8","length":"a"},{"start":"cc","length":"a"},{"start":"e0","length":"a"},{"start":"f4","length":"a"},{"start":"108","length":"3"}]},{"pgid":"2.0","version":"87'117","reported_seq":931,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-03T17:50:23.294028+0000","last_change":"2023-10-03T17:40:10.708344+0000","last_active":"2023-10-03T17:50:23.294028+0000","last_peered":"2023-10-03T17:50:23.294028+0000","last_clean":"2023-10-03T17:50:23.294028+0000","last_became_active":"2023-10-03T17:34:42.272306+0000","last_became_peered":"2023-10-03T17:34:42.272306+0000","last_unstale":"2023-10-03T17:50:23.294028+0000","last_undegraded":"2023-10-03T17:50:23.294028+0000","last_fullsized":"2023-10-03T17:50:23.294028+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'20","last_scrub_stamp":"2023-10-03T17:40:10.708060+0000","last_deep_scrub":"46'20","last_deep_scrub_stamp":"2023-10-03T17:40:10.708060+0000","last_clean_scrub_stamp":"2023-10-03T17:40:10.708060+0000","objects_scrubbed":7,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:03:09.097252+0000","scrub_duration":0.042279491000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":653,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":2179,"num_read_kb":1786,"num_write":117,"num_write_kb":89,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":301,"num_omap_keys":10,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"88'114229","reported_seq":187403,"reported_epoch":88,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T17:50:27.924649+0000","last_change":"2023-10-03T17:50:27.747258+0000","last_active":"2023-10-03T17:50:27.924649+0000","last_peered":"2023-10-03T17:50:27.924649+0000","last_clean":"2023-10-03T17:50:27.924649+0000","last_became_active":"2023-10-03T17:34:48.310612+0000","last_became_peered":"2023-10-03T17:34:48.310612+0000","last_unstale":"2023-10-03T17:50:27.924649+0000","last_undegraded":"2023-10-03T17:50:27.924649+0000","last_fullsized":"2023-10-03T17:50:27.924649+0000","mapping_epoch":16,"log_start":"79'104171","ondisk_log_start":"79'104171","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'75807","last_scrub_stamp":"2023-10-03T17:45:43.833310+0000","last_deep_scrub":"61'75807","last_deep_scrub_stamp":"2023-10-03T17:45:43.833310+0000","last_clean_scrub_stamp":"2023-10-03T17:45:43.833310+0000","objects_scrubbed":3607,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":33,"last_scrub_duration":315,"scrub_schedule":"queued for deep scrub","scrub_duration":314.26374841099999,"objects_trimmed":304,"snaptrim_duration":18.305336861000001,"stat_sum":{"num_bytes":5519212563,"num_objects":8136,"num_object_clones":6848,"num_object_copies":16272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8136,"num_whiteouts":0,"num_read":87374,"num_read_kb":330088,"num_write":88768,"num_write_kb":355066,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"54","length":"1e"},{"start":"7c","length":"8"},{"start":"8a","length":"2"},{"start":"90","length":"a"},{"start":"a4","length":"a"},{"start":"b8","length":"a"},{"start":"cc","length":"a"},{"start":"e0","length":"7"}]},{"pgid":"2.3","version":"82'47","reported_seq":178,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-03T17:50:08.417278+0000","last_change":"2023-10-03T17:40:24.472734+0000","last_active":"2023-10-03T17:50:08.417278+0000","last_peered":"2023-10-03T17:50:08.417278+0000","last_clean":"2023-10-03T17:50:08.417278+0000","last_became_active":"2023-10-03T17:34:42.270194+0000","last_became_peered":"2023-10-03T17:34:42.270194+0000","last_unstale":"2023-10-03T17:50:08.417278+0000","last_undegraded":"2023-10-03T17:50:08.417278+0000","last_fullsized":"2023-10-03T17:50:08.417278+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'8","last_scrub_stamp":"2023-10-03T17:40:24.472573+0000","last_deep_scrub":"46'8","last_deep_scrub_stamp":"2023-10-03T17:40:24.472573+0000","last_clean_scrub_stamp":"2023-10-03T17:40:24.472573+0000","objects_scrubbed":4,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T18:28:04.735033+0000","scrub_duration":0.024990637999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":385,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":43,"num_read_kb":25,"num_write":43,"num_write_kb":26,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"46'99","reported_seq":232,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-03T17:50:08.429267+0000","last_change":"2023-10-03T17:40:10.452341+0000","last_active":"2023-10-03T17:50:08.429267+0000","last_peered":"2023-10-03T17:50:08.429267+0000","last_clean":"2023-10-03T17:50:08.429267+0000","last_became_active":"2023-10-03T17:34:39.502393+0000","last_became_peered":"2023-10-03T17:34:39.502393+0000","last_unstale":"2023-10-03T17:50:08.429267+0000","last_undegraded":"2023-10-03T17:50:08.429267+0000","last_fullsized":"2023-10-03T17:50:08.429267+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'99","last_scrub_stamp":"2023-10-03T17:40:10.449099+0000","last_deep_scrub":"46'99","last_deep_scrub_stamp":"2023-10-03T17:40:10.449099+0000","last_clean_scrub_stamp":"2023-10-03T17:40:10.449099+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T20:18:04.958666+0000","scrub_duration":0.043225012,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":21955178515,"num_objects":37618,"num_object_clones":32497,"num_object_copies":75236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37618,"num_whiteouts":0,"num_read":348631,"num_read_kb":1316176,"num_write":353104,"num_write_kb":1412410,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2893307904,"data_stored":15860641830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40138,"ondisk_log_size":40138,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":5372,"num_objects":143,"num_object_clones":0,"num_object_copies":286,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":143,"num_whiteouts":0,"num_read":45749,"num_read_kb":38222,"num_write":1843,"num_write_kb":1483,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":983040,"data_stored":10744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7029,"internal_metadata":0},"log_size":10692,"ondisk_log_size":10692,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":7460,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738564,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1993288,"kb_used_data":1412992,"kb_used_omap":2891,"kb_used_meta":577396,"kb_avail":92378552,"statfs":{"total":96636764160,"available":94595637248,"internally_reserved":0,"allocated":1446903808,"data_stored":7924785134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2960945,"internal_metadata":591253967},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,6,4,4,10,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 17:50:02 2023","interfaces":[{"interface":"back","average":{"1min":1.2230000000000001,"5min":0.60599999999999998,"15min":0.48699999999999999},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.14299999999999999},"max":{"1min":14.787000000000001,"5min":14.787000000000001,"15min":14.787000000000001},"last":0.188},{"interface":"front","average":{"1min":1.4319999999999999,"5min":0.60899999999999999,"15min":0.48899999999999999},"min":{"1min":0.185,"5min":0.17999999999999999,"15min":0.17100000000000001},"max":{"1min":14.919,"5min":14.919,"15min":14.919},"last":0.745}]},{"osd":1,"last update":"Tue Oct 3 17:50:02 2023","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.52400000000000002,"15min":0.51400000000000001},"min":{"1min":0.221,"5min":0.184,"15min":0.16600000000000001},"max":{"1min":6.7539999999999996,"5min":6.7539999999999996,"15min":8.1059999999999999},"last":0.218},{"interface":"front","average":{"1min":0.89800000000000002,"5min":0.57899999999999996,"15min":0.47599999999999998},"min":{"1min":0.22700000000000001,"5min":0.223,"15min":0.17699999999999999},"max":{"1min":5.194,"5min":5.194,"15min":5.194},"last":0.78100000000000003}]}]},{"osd":1,"up_from":8,"seq":34359738562,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1884928,"kb_used_data":696120,"kb_used_omap":1,"kb_used_meta":1188798,"kb_avail":92486912,"statfs":{"total":96636764160,"available":94706597888,"internally_reserved":0,"allocated":712826880,"data_stored":4467663193,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":1217329606},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,0,3,7,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 17:50:17 2023","interfaces":[{"interface":"back","average":{"1min":0.33000000000000002,"5min":0.33500000000000002,"15min":0.35199999999999998},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.161},"max":{"1min":0.84899999999999998,"5min":2.7050000000000001,"15min":4.1879999999999997},"last":0.27300000000000002},{"interface":"front","average":{"1min":0.313,"5min":0.39100000000000001,"15min":0.39700000000000002},"min":{"1min":0.21199999999999999,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.53400000000000003,"5min":5.8959999999999999,"15min":5.8959999999999999},"last":0.33100000000000002}]},{"osd":2,"last update":"Tue Oct 3 17:50:17 2023","interfaces":[{"interface":"back","average":{"1min":0.39700000000000002,"5min":0.47999999999999998,"15min":0.41899999999999998},"min":{"1min":0.223,"5min":0.17599999999999999,"15min":0.159},"max":{"1min":1.391,"5min":4.5519999999999996,"15min":4.5519999999999996},"last":0.215},{"interface":"front","average":{"1min":0.40699999999999997,"5min":0.42899999999999999,"15min":0.39300000000000002},"min":{"1min":0.27500000000000002,"5min":0.23200000000000001,"15min":0.16400000000000001},"max":{"1min":0.91100000000000003,"5min":2.6970000000000001,"15min":3.0699999999999998},"last":0.307}]}]},{"osd":0,"up_from":8,"seq":34359738562,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1923460,"kb_used_data":717884,"kb_used_omap":1,"kb_used_meta":1205566,"kb_avail":92448380,"statfs":{"total":96636764160,"available":94667141120,"internally_reserved":0,"allocated":735113216,"data_stored":3452103573,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":1234500038},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,1,1,7,8,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 17:50:03 2023","interfaces":[{"interface":"back","average":{"1min":0.29499999999999998,"5min":0.38100000000000001,"15min":0.40500000000000003},"min":{"1min":0.17999999999999999,"5min":0.17499999999999999,"15min":0.154},"max":{"1min":0.48999999999999999,"5min":3.4649999999999999,"15min":5.7649999999999997},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.46999999999999997,"15min":0.46700000000000003},"min":{"1min":0.23999999999999999,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":3.1320000000000001,"5min":3.4279999999999999,"15min":9.4770000000000003},"last":0.313}]},{"osd":2,"last update":"Tue Oct 3 17:50:03 2023","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.54600000000000004,"15min":0.51400000000000001},"min":{"1min":0.25,"5min":0.20000000000000001,"15min":0.19900000000000001},"max":{"1min":3.2149999999999999,"5min":4.1619999999999999,"15min":9.4149999999999991},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.38,"5min":0.55700000000000005,"15min":0.48399999999999999},"min":{"1min":0.23899999999999999,"5min":0.224,"15min":0.189},"max":{"1min":0.73699999999999999,"5min":5.218,"15min":5.218},"last":0.379}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":356352,"data_stored":3915,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":5372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":135168,"data_stored":1457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7029,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":734126080,"data_stored":3452772352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":712200192,"data_stored":4474429459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1446981632,"data_stored":7933440019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-03T17:50:32.632 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-03T17:34:47.281095+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=34, tm_sec=47, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=40, tm_sec=6, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T17:50:32.632 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.0 2023-10-03T17:50:33.245 INFO:teuthology.orchestra.run.smithi111.stderr:instructing pg 3.0 on osd.1 to deep-scrub 2023-10-03T17:50:33.279 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T17:50:53.280 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T17:50:54.478 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-03T17:50:54.478 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-10-03T17:50:54.526 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":575,"stamp":"2023-10-03T17:50:54.192625+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":21991892783,"num_objects":37383,"num_object_clones":32117,"num_object_copies":74766,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37383,"num_whiteouts":0,"num_read":402157,"num_read_kb":1384617,"num_write":363045,"num_write_kb":1447631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":51374,"ondisk_log_size":51374,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":5995420,"kb_used_data":2890308,"kb_used_omap":2894,"kb_used_meta":3102193,"kb_avail":277120100,"statfs":{"total":289910292480,"available":283770982400,"internally_reserved":0,"allocated":2959675392,"data_stored":15797173468,"data_compressed":7460,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":2964133,"internal_metadata":3176645979},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,2,3,12,9,13,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":22593536,"num_objects":-386,"num_object_clones":-386,"num_object_copies":-772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-386,"num_whiteouts":0,"num_read":4751,"num_read_kb":18692,"num_write":4928,"num_write_kb":19712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002122"},"pg_stats":[{"pgid":"2.7","version":"88'1959","reported_seq":4131,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-03T17:50:51.688359+0000","last_change":"2023-10-03T17:40:17.407011+0000","last_active":"2023-10-03T17:50:51.688359+0000","last_peered":"2023-10-03T17:50:51.688359+0000","last_clean":"2023-10-03T17:50:51.688359+0000","last_became_active":"2023-10-03T17:34:42.276624+0000","last_became_peered":"2023-10-03T17:34:42.276624+0000","last_unstale":"2023-10-03T17:50:51.688359+0000","last_undegraded":"2023-10-03T17:50:51.688359+0000","last_fullsized":"2023-10-03T17:50:51.688359+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'673","last_scrub_stamp":"2023-10-03T17:40:17.406813+0000","last_deep_scrub":"46'673","last_deep_scrub_stamp":"2023-10-03T17:40:17.406813+0000","last_clean_scrub_stamp":"2023-10-03T17:40:17.406813+0000","objects_scrubbed":7,"log_size":1959,"log_dups_size":0,"ondisk_log_size":1959,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T21:38:35.056889+0000","scrub_duration":0.013778076,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":789,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":6411,"num_read_kb":5295,"num_write":263,"num_write_kb":209,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":903,"num_omap_keys":30,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"88'3248","reported_seq":8268,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-03T17:50:51.688435+0000","last_change":"2023-10-03T17:40:13.376519+0000","last_active":"2023-10-03T17:50:51.688435+0000","last_peered":"2023-10-03T17:50:51.688435+0000","last_clean":"2023-10-03T17:50:51.688435+0000","last_became_active":"2023-10-03T17:34:42.271201+0000","last_became_peered":"2023-10-03T17:34:42.271201+0000","last_unstale":"2023-10-03T17:50:51.688435+0000","last_undegraded":"2023-10-03T17:50:51.688435+0000","last_fullsized":"2023-10-03T17:50:51.688435+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'1129","last_scrub_stamp":"2023-10-03T17:40:13.376332+0000","last_deep_scrub":"46'1129","last_deep_scrub_stamp":"2023-10-03T17:40:13.376332+0000","last_clean_scrub_stamp":"2023-10-03T17:40:13.376332+0000","objects_scrubbed":11,"log_size":3248,"log_dups_size":0,"ondisk_log_size":3248,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T18:59:17.088585+0000","scrub_duration":0.015998675,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":723,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":12741,"num_read_kb":10887,"num_write":446,"num_write_kb":367,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2912,"num_omap_keys":93,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"85'50","reported_seq":182,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-03T17:50:08.417281+0000","last_change":"2023-10-03T17:40:25.496010+0000","last_active":"2023-10-03T17:50:08.417281+0000","last_peered":"2023-10-03T17:50:08.417281+0000","last_clean":"2023-10-03T17:50:08.417281+0000","last_became_active":"2023-10-03T17:34:42.276941+0000","last_became_peered":"2023-10-03T17:34:42.276941+0000","last_unstale":"2023-10-03T17:50:08.417281+0000","last_undegraded":"2023-10-03T17:50:08.417281+0000","last_fullsized":"2023-10-03T17:50:08.417281+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'14","last_scrub_stamp":"2023-10-03T17:40:25.495827+0000","last_deep_scrub":"46'14","last_deep_scrub_stamp":"2023-10-03T17:40:25.495827+0000","last_clean_scrub_stamp":"2023-10-03T17:40:25.495827+0000","objects_scrubbed":7,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:05:44.939663+0000","scrub_duration":0.0133078,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":957,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":50,"num_read_kb":23,"num_write":43,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"88'2596","reported_seq":6219,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-03T17:50:51.686251+0000","last_change":"2023-10-03T17:40:26.513902+0000","last_active":"2023-10-03T17:50:51.686251+0000","last_peered":"2023-10-03T17:50:51.686251+0000","last_clean":"2023-10-03T17:50:51.686251+0000","last_became_active":"2023-10-03T17:34:42.276459+0000","last_became_peered":"2023-10-03T17:34:42.276459+0000","last_unstale":"2023-10-03T17:50:51.686251+0000","last_undegraded":"2023-10-03T17:50:51.686251+0000","last_fullsized":"2023-10-03T17:50:51.686251+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'964","last_scrub_stamp":"2023-10-03T17:40:26.513729+0000","last_deep_scrub":"46'964","last_deep_scrub_stamp":"2023-10-03T17:40:26.513729+0000","last_clean_scrub_stamp":"2023-10-03T17:40:26.513729+0000","objects_scrubbed":10,"log_size":2596,"log_dups_size":0,"ondisk_log_size":2596,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:41:33.182751+0000","scrub_duration":0.018190280999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":537,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":9369,"num_read_kb":7834,"num_write":330,"num_write_kb":274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1731,"num_omap_keys":80,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"88'118338","reported_seq":195606,"reported_epoch":88,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T17:50:53.111894+0000","last_change":"2023-10-03T17:50:52.317722+0000","last_active":"2023-10-03T17:50:53.111894+0000","last_peered":"2023-10-03T17:50:53.111894+0000","last_clean":"2023-10-03T17:50:53.111894+0000","last_became_active":"2023-10-03T17:34:48.310701+0000","last_became_peered":"2023-10-03T17:34:48.310701+0000","last_unstale":"2023-10-03T17:50:53.111894+0000","last_undegraded":"2023-10-03T17:50:53.111894+0000","last_fullsized":"2023-10-03T17:50:53.111894+0000","mapping_epoch":16,"log_start":"79'108249","ondisk_log_start":"79'108249","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'80695","last_scrub_stamp":"2023-10-03T17:46:13.982571+0000","last_deep_scrub":"66'80695","last_deep_scrub_stamp":"2023-10-03T17:46:13.982571+0000","last_clean_scrub_stamp":"2023-10-03T17:46:13.982571+0000","objects_scrubbed":3410,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":42,"last_scrub_duration":363,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:57:32.446767+0000","scrub_duration":362.26574615599998,"objects_trimmed":714,"snaptrim_duration":42.895739675999998,"stat_sum":{"num_bytes":5668638720,"num_objects":9199,"num_object_clones":7879,"num_object_copies":18398,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9199,"num_whiteouts":0,"num_read":91870,"num_read_kb":346984,"num_write":93383,"num_write_kb":373532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"54","length":"1e"},{"start":"7c","length":"8"},{"start":"8a","length":"2"},{"start":"90","length":"a"},{"start":"a4","length":"a"},{"start":"b8","length":"a"},{"start":"cc","length":"8"}]},{"pgid":"2.2","version":"88'2701","reported_seq":5592,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-03T17:50:52.941193+0000","last_change":"2023-10-03T17:40:08.675119+0000","last_active":"2023-10-03T17:50:52.941193+0000","last_peered":"2023-10-03T17:50:52.941193+0000","last_clean":"2023-10-03T17:50:52.941193+0000","last_became_active":"2023-10-03T17:34:42.270625+0000","last_became_peered":"2023-10-03T17:34:42.270625+0000","last_unstale":"2023-10-03T17:50:52.941193+0000","last_undegraded":"2023-10-03T17:50:52.941193+0000","last_fullsized":"2023-10-03T17:50:52.941193+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'841","last_scrub_stamp":"2023-10-03T17:40:08.674910+0000","last_deep_scrub":"46'841","last_deep_scrub_stamp":"2023-10-03T17:40:08.674910+0000","last_clean_scrub_stamp":"2023-10-03T17:40:08.674910+0000","objects_scrubbed":11,"log_size":2701,"log_dups_size":0,"ondisk_log_size":2701,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T21:05:42.790614+0000","scrub_duration":0.039189555000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":909,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":8572,"num_read_kb":7095,"num_write":360,"num_write_kb":292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2564,"num_omap_keys":60,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"88'105190","reported_seq":180123,"reported_epoch":88,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T17:50:53.111027+0000","last_change":"2023-10-03T17:43:28.494717+0000","last_active":"2023-10-03T17:50:53.111027+0000","last_peered":"2023-10-03T17:50:53.111027+0000","last_clean":"2023-10-03T17:50:53.111027+0000","last_became_active":"2023-10-03T17:34:48.310555+0000","last_became_peered":"2023-10-03T17:34:48.310555+0000","last_unstale":"2023-10-03T17:50:53.111027+0000","last_undegraded":"2023-10-03T17:50:53.111027+0000","last_fullsized":"2023-10-03T17:50:53.111027+0000","mapping_epoch":16,"log_start":"79'95116","ondisk_log_start":"79'95116","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_clean_scrub_stamp":"2023-10-03T17:34:47.281095+0000","objects_scrubbed":6976,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":120,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 444s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086289699999999995,"stat_sum":{"num_bytes":5333041152,"num_objects":13377,"num_object_clones":12135,"num_object_copies":26754,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13377,"num_whiteouts":0,"num_read":86479,"num_read_kb":325900,"num_write":87063,"num_write_kb":348252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"88'226","reported_seq":2379,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-03T17:50:48.891589+0000","last_change":"2023-10-03T17:40:17.641493+0000","last_active":"2023-10-03T17:50:48.891589+0000","last_peered":"2023-10-03T17:50:48.891589+0000","last_clean":"2023-10-03T17:50:48.891589+0000","last_became_active":"2023-10-03T17:34:42.278196+0000","last_became_peered":"2023-10-03T17:34:42.278196+0000","last_unstale":"2023-10-03T17:50:48.891589+0000","last_undegraded":"2023-10-03T17:50:48.891589+0000","last_fullsized":"2023-10-03T17:50:48.891589+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'30","last_scrub_stamp":"2023-10-03T17:40:17.640969+0000","last_deep_scrub":"46'30","last_deep_scrub_stamp":"2023-10-03T17:40:17.640969+0000","last_clean_scrub_stamp":"2023-10-03T17:40:17.640969+0000","objects_scrubbed":6,"log_size":226,"log_dups_size":0,"ondisk_log_size":226,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:04:02.954896+0000","scrub_duration":0.071054565,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":419,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":6384,"num_read_kb":5277,"num_write":241,"num_write_kb":194,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":903,"num_omap_keys":30,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"88'117188","reported_seq":190735,"reported_epoch":88,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T17:50:53.108419+0000","last_change":"2023-10-03T17:50:51.221745+0000","last_active":"2023-10-03T17:50:53.108419+0000","last_peered":"2023-10-03T17:50:53.108419+0000","last_clean":"2023-10-03T17:50:53.108419+0000","last_became_active":"2023-10-03T17:34:48.310912+0000","last_became_peered":"2023-10-03T17:34:48.310912+0000","last_unstale":"2023-10-03T17:50:53.108419+0000","last_undegraded":"2023-10-03T17:50:53.108419+0000","last_fullsized":"2023-10-03T17:50:53.108419+0000","mapping_epoch":16,"log_start":"79'107105","ondisk_log_start":"79'107105","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'58196","last_scrub_stamp":"2023-10-03T17:43:26.193075+0000","last_deep_scrub":"53'58196","last_deep_scrub_stamp":"2023-10-03T17:43:26.193075+0000","last_clean_scrub_stamp":"2023-10-03T17:43:26.193075+0000","objects_scrubbed":2326,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":8,"last_scrub_duration":196,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:16:36.461572+0000","scrub_duration":195.44739730800001,"objects_trimmed":826,"snaptrim_duration":41.782548759000001,"stat_sum":{"num_bytes":5461696512,"num_objects":6736,"num_object_clones":5465,"num_object_copies":13472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6736,"num_whiteouts":0,"num_read":88599,"num_read_kb":335528,"num_write":89822,"num_write_kb":359288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"54","length":"1e"},{"start":"7c","length":"8"},{"start":"8a","length":"2"},{"start":"90","length":"a"},{"start":"a4","length":"a"},{"start":"b8","length":"a"},{"start":"cc","length":"a"},{"start":"e0","length":"a"},{"start":"f4","length":"a"},{"start":"108","length":"a"},{"start":"11c","length":"2"}]},{"pgid":"2.0","version":"87'117","reported_seq":936,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-03T17:50:48.600360+0000","last_change":"2023-10-03T17:40:10.708344+0000","last_active":"2023-10-03T17:50:48.600360+0000","last_peered":"2023-10-03T17:50:48.600360+0000","last_clean":"2023-10-03T17:50:48.600360+0000","last_became_active":"2023-10-03T17:34:42.272306+0000","last_became_peered":"2023-10-03T17:34:42.272306+0000","last_unstale":"2023-10-03T17:50:48.600360+0000","last_undegraded":"2023-10-03T17:50:48.600360+0000","last_fullsized":"2023-10-03T17:50:48.600360+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'20","last_scrub_stamp":"2023-10-03T17:40:10.708060+0000","last_deep_scrub":"46'20","last_deep_scrub_stamp":"2023-10-03T17:40:10.708060+0000","last_clean_scrub_stamp":"2023-10-03T17:40:10.708060+0000","objects_scrubbed":7,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:03:09.097252+0000","scrub_duration":0.042279491000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":653,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":2179,"num_read_kb":1786,"num_write":117,"num_write_kb":89,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":301,"num_omap_keys":10,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"88'117473","reported_seq":192157,"reported_epoch":88,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T17:50:53.075843+0000","last_change":"2023-10-03T17:50:50.357821+0000","last_active":"2023-10-03T17:50:53.075843+0000","last_peered":"2023-10-03T17:50:53.075843+0000","last_clean":"2023-10-03T17:50:53.075843+0000","last_became_active":"2023-10-03T17:34:48.310612+0000","last_became_peered":"2023-10-03T17:34:48.310612+0000","last_unstale":"2023-10-03T17:50:53.075843+0000","last_undegraded":"2023-10-03T17:50:53.075843+0000","last_fullsized":"2023-10-03T17:50:53.075843+0000","mapping_epoch":16,"log_start":"79'107388","ondisk_log_start":"79'107388","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'75807","last_scrub_stamp":"2023-10-03T17:45:43.833310+0000","last_deep_scrub":"61'75807","last_deep_scrub_stamp":"2023-10-03T17:45:43.833310+0000","last_clean_scrub_stamp":"2023-10-03T17:45:43.833310+0000","objects_scrubbed":3607,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":26,"last_scrub_duration":315,"scrub_schedule":"queued for deep scrub","scrub_duration":314.26374841099999,"objects_trimmed":779,"snaptrim_duration":40.901849300000002,"stat_sum":{"num_bytes":5527920659,"num_objects":7926,"num_object_clones":6638,"num_object_copies":15852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7926,"num_whiteouts":0,"num_read":89354,"num_read_kb":337892,"num_write":90797,"num_write_kb":363182,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"54","length":"1e"},{"start":"7c","length":"8"},{"start":"8a","length":"2"},{"start":"90","length":"a"},{"start":"a4","length":"a"},{"start":"b8","length":"a"},{"start":"cc","length":"a"},{"start":"e0","length":"a"},{"start":"f4","length":"4"}]},{"pgid":"2.3","version":"82'47","reported_seq":178,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-03T17:50:08.417278+0000","last_change":"2023-10-03T17:40:24.472734+0000","last_active":"2023-10-03T17:50:08.417278+0000","last_peered":"2023-10-03T17:50:08.417278+0000","last_clean":"2023-10-03T17:50:08.417278+0000","last_became_active":"2023-10-03T17:34:42.270194+0000","last_became_peered":"2023-10-03T17:34:42.270194+0000","last_unstale":"2023-10-03T17:50:08.417278+0000","last_undegraded":"2023-10-03T17:50:08.417278+0000","last_fullsized":"2023-10-03T17:50:08.417278+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'8","last_scrub_stamp":"2023-10-03T17:40:24.472573+0000","last_deep_scrub":"46'8","last_deep_scrub_stamp":"2023-10-03T17:40:24.472573+0000","last_clean_scrub_stamp":"2023-10-03T17:40:24.472573+0000","objects_scrubbed":4,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T18:28:04.735033+0000","scrub_duration":0.024990637999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":385,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":43,"num_read_kb":25,"num_write":43,"num_write_kb":26,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"46'99","reported_seq":233,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-03T17:50:09.442198+0000","last_change":"2023-10-03T17:40:10.452341+0000","last_active":"2023-10-03T17:50:09.442198+0000","last_peered":"2023-10-03T17:50:09.442198+0000","last_clean":"2023-10-03T17:50:09.442198+0000","last_became_active":"2023-10-03T17:34:39.502393+0000","last_became_peered":"2023-10-03T17:34:39.502393+0000","last_unstale":"2023-10-03T17:50:09.442198+0000","last_undegraded":"2023-10-03T17:50:09.442198+0000","last_fullsized":"2023-10-03T17:50:09.442198+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'99","last_scrub_stamp":"2023-10-03T17:40:10.449099+0000","last_deep_scrub":"46'99","last_deep_scrub_stamp":"2023-10-03T17:40:10.449099+0000","last_clean_scrub_stamp":"2023-10-03T17:40:10.449099+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T20:18:04.958666+0000","scrub_duration":0.043225012,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":21991297043,"num_objects":37238,"num_object_clones":32117,"num_object_copies":74476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37238,"num_whiteouts":0,"num_read":356302,"num_read_kb":1346304,"num_write":361065,"num_write_kb":1444254,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2958508032,"data_stored":15784210470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40331,"ondisk_log_size":40331,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":5372,"num_objects":143,"num_object_clones":0,"num_object_copies":286,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":143,"num_whiteouts":0,"num_read":45749,"num_read_kb":38222,"num_write":1843,"num_write_kb":1483,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":983040,"data_stored":10744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7029,"internal_metadata":0},"log_size":10944,"ondisk_log_size":10944,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":7460,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738569,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2097004,"kb_used_data":1444580,"kb_used_omap":2891,"kb_used_meta":649524,"kb_avail":92274836,"statfs":{"total":96636764160,"available":94489432064,"internally_reserved":0,"allocated":1479249920,"data_stored":7897681902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2960945,"internal_metadata":665113039},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,3,6,7,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 17:50:02 2023","interfaces":[{"interface":"back","average":{"1min":1.2230000000000001,"5min":0.60599999999999998,"15min":0.48699999999999999},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.14299999999999999},"max":{"1min":14.787000000000001,"5min":14.787000000000001,"15min":14.787000000000001},"last":0.26900000000000002},{"interface":"front","average":{"1min":1.4319999999999999,"5min":0.60899999999999999,"15min":0.48899999999999999},"min":{"1min":0.185,"5min":0.17999999999999999,"15min":0.17100000000000001},"max":{"1min":14.919,"5min":14.919,"15min":14.919},"last":0.253}]},{"osd":1,"last update":"Tue Oct 3 17:50:02 2023","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.52400000000000002,"15min":0.51400000000000001},"min":{"1min":0.221,"5min":0.184,"15min":0.16600000000000001},"max":{"1min":6.7539999999999996,"5min":6.7539999999999996,"15min":8.1059999999999999},"last":0.32300000000000001},{"interface":"front","average":{"1min":0.89800000000000002,"5min":0.57899999999999996,"15min":0.47599999999999998},"min":{"1min":0.22700000000000001,"5min":0.223,"15min":0.17699999999999999},"max":{"1min":5.194,"5min":5.194,"15min":5.194},"last":0.34000000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738567,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1936968,"kb_used_data":712064,"kb_used_omap":1,"kb_used_meta":1224894,"kb_avail":92434872,"statfs":{"total":96636764160,"available":94653308928,"internally_reserved":0,"allocated":729153536,"data_stored":4524999001,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":1254291910},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,5,0,4,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 17:50:17 2023","interfaces":[{"interface":"back","average":{"1min":0.33000000000000002,"5min":0.33500000000000002,"15min":0.35199999999999998},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.161},"max":{"1min":0.84899999999999998,"5min":2.7050000000000001,"15min":4.1879999999999997},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.313,"5min":0.39100000000000001,"15min":0.39700000000000002},"min":{"1min":0.21199999999999999,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.53400000000000003,"5min":5.8959999999999999,"15min":5.8959999999999999},"last":0.53100000000000003}]},{"osd":2,"last update":"Tue Oct 3 17:50:17 2023","interfaces":[{"interface":"back","average":{"1min":0.39700000000000002,"5min":0.47999999999999998,"15min":0.41899999999999998},"min":{"1min":0.223,"5min":0.17599999999999999,"15min":0.159},"max":{"1min":1.391,"5min":4.5519999999999996,"15min":4.5519999999999996},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.40699999999999997,"5min":0.42899999999999999,"15min":0.39300000000000002},"min":{"1min":0.27500000000000002,"5min":0.23200000000000001,"15min":0.16400000000000001},"max":{"1min":0.91100000000000003,"5min":2.6970000000000001,"15min":3.0699999999999998},"last":0.39500000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738567,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1961448,"kb_used_data":733664,"kb_used_omap":1,"kb_used_meta":1227774,"kb_avail":92410392,"statfs":{"total":96636764160,"available":94628241408,"internally_reserved":0,"allocated":751271936,"data_stored":3374492565,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":1257241030},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,0,4,3,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 17:50:03 2023","interfaces":[{"interface":"back","average":{"1min":0.29499999999999998,"5min":0.38100000000000001,"15min":0.40500000000000003},"min":{"1min":0.17999999999999999,"5min":0.17499999999999999,"15min":0.154},"max":{"1min":0.48999999999999999,"5min":3.4649999999999999,"15min":5.7649999999999997},"last":0.23499999999999999},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.46999999999999997,"15min":0.46700000000000003},"min":{"1min":0.23999999999999999,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":3.1320000000000001,"5min":3.4279999999999999,"15min":9.4770000000000003},"last":0.253}]},{"osd":2,"last update":"Tue Oct 3 17:50:03 2023","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.54600000000000004,"15min":0.51400000000000001},"min":{"1min":0.25,"5min":0.20000000000000001,"15min":0.19900000000000001},"max":{"1min":3.2149999999999999,"5min":4.1619999999999999,"15min":9.4149999999999991},"last":0.20499999999999999},{"interface":"front","average":{"1min":0.38,"5min":0.55700000000000005,"15min":0.48399999999999999},"min":{"1min":0.23899999999999999,"5min":0.224,"15min":0.189},"max":{"1min":0.73699999999999999,"5min":5.218,"15min":5.218},"last":0.217}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":356352,"data_stored":3915,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":5372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":135168,"data_stored":1457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7029,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":750419968,"data_stored":3369488384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":728530944,"data_stored":4524007443,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1479557120,"data_stored":7890714643,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-03T17:50:54.528 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-03T17:34:47.281095+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=34, tm_sec=47, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=40, tm_sec=6, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T17:50:54.528 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T17:51:07.078 INFO:tasks.ceph.osd.0.smithi111.stderr:2023-10-03T17:51:07.078+0000 7ffbac289640 -1 reset not still connected to 0x5563a7ff68f0 2023-10-03T17:51:07.079 INFO:tasks.ceph.osd.0.smithi111.stderr:2023-10-03T17:51:07.078+0000 7ffbac289640 -1 reset not still connected to 0x5563a838e1a0 2023-10-03T17:51:08.789 INFO:tasks.ceph.osd.0.smithi111.stderr:2023-10-03T17:51:08.790+0000 7ffbac289640 -1 reset not still connected to 0x5563a7ff68f0 2023-10-03T17:51:08.789 INFO:tasks.ceph.osd.0.smithi111.stderr:2023-10-03T17:51:08.790+0000 7ffbac289640 -1 reset not still connected to 0x5563a838e1a0 2023-10-03T17:51:08.789 INFO:tasks.ceph.osd.0.smithi111.stderr:2023-10-03T17:51:08.790+0000 7ffbac289640 -1 reset not still connected to 0x5563a851aea0 2023-10-03T17:51:14.530 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T17:51:15.235 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-03T17:51:15.235 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-10-03T17:51:15.289 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":590,"stamp":"2023-10-03T17:51:14.195767+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":22017480495,"num_objects":38683,"num_object_clones":33417,"num_object_copies":77366,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38683,"num_whiteouts":0,"num_read":410979,"num_read_kb":1410157,"num_write":368865,"num_write_kb":1470191,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":51538,"ondisk_log_size":51538,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":6250048,"kb_used_data":2936744,"kb_used_omap":2894,"kb_used_meta":3310385,"kb_avail":276865472,"statfs":{"total":289910292480,"available":283510243328,"internally_reserved":0,"allocated":3007225856,"data_stored":16672740555,"data_compressed":7460,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":2964133,"internal_metadata":3389834587},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,4,1,7,13,14,34,13],"upper_bound":512},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":11427789,"num_objects":1340,"num_object_clones":1341,"num_object_copies":2680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1340,"num_whiteouts":0,"num_read":2894,"num_read_kb":10737,"num_write":2578,"num_write_kb":10167,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001535"},"pg_stats":[{"pgid":"2.7","version":"93'2027","reported_seq":4321,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-10-03T17:51:10.679522+0000","last_change":"2023-10-03T17:40:17.407011+0000","last_active":"2023-10-03T17:51:10.679522+0000","last_peered":"2023-10-03T17:51:10.679522+0000","last_clean":"2023-10-03T17:51:10.679522+0000","last_became_active":"2023-10-03T17:34:42.276624+0000","last_became_peered":"2023-10-03T17:34:42.276624+0000","last_unstale":"2023-10-03T17:51:10.679522+0000","last_undegraded":"2023-10-03T17:51:10.679522+0000","last_fullsized":"2023-10-03T17:51:10.679522+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'673","last_scrub_stamp":"2023-10-03T17:40:17.406813+0000","last_deep_scrub":"46'673","last_deep_scrub_stamp":"2023-10-03T17:40:17.406813+0000","last_clean_scrub_stamp":"2023-10-03T17:40:17.406813+0000","objects_scrubbed":7,"log_size":2027,"log_dups_size":0,"ondisk_log_size":2027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T21:38:35.056889+0000","scrub_duration":0.013778076,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":690,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":6845,"num_read_kb":5657,"num_write":298,"num_write_kb":233,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":903,"num_omap_keys":30,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"93'3353","reported_seq":8649,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-10-03T17:51:10.685902+0000","last_change":"2023-10-03T17:40:13.376519+0000","last_active":"2023-10-03T17:51:10.685902+0000","last_peered":"2023-10-03T17:51:10.685902+0000","last_clean":"2023-10-03T17:51:10.685902+0000","last_became_active":"2023-10-03T17:34:42.271201+0000","last_became_peered":"2023-10-03T17:34:42.271201+0000","last_unstale":"2023-10-03T17:51:10.685902+0000","last_undegraded":"2023-10-03T17:51:10.685902+0000","last_fullsized":"2023-10-03T17:51:10.685902+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'1129","last_scrub_stamp":"2023-10-03T17:40:13.376332+0000","last_deep_scrub":"46'1129","last_deep_scrub_stamp":"2023-10-03T17:40:13.376332+0000","last_clean_scrub_stamp":"2023-10-03T17:40:13.376332+0000","objects_scrubbed":11,"log_size":3353,"log_dups_size":0,"ondisk_log_size":3353,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T18:59:17.088585+0000","scrub_duration":0.015998675,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":13539,"num_read_kb":11565,"num_write":496,"num_write_kb":404,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2912,"num_omap_keys":93,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"91'53","reported_seq":191,"reported_epoch":92,"state":"active+clean","last_fresh":"2023-10-03T17:51:03.434452+0000","last_change":"2023-10-03T17:40:25.496010+0000","last_active":"2023-10-03T17:51:03.434452+0000","last_peered":"2023-10-03T17:51:03.434452+0000","last_clean":"2023-10-03T17:51:03.434452+0000","last_became_active":"2023-10-03T17:34:42.276941+0000","last_became_peered":"2023-10-03T17:34:42.276941+0000","last_unstale":"2023-10-03T17:51:03.434452+0000","last_undegraded":"2023-10-03T17:51:03.434452+0000","last_fullsized":"2023-10-03T17:51:03.434452+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'14","last_scrub_stamp":"2023-10-03T17:40:25.495827+0000","last_deep_scrub":"46'14","last_deep_scrub_stamp":"2023-10-03T17:40:25.495827+0000","last_clean_scrub_stamp":"2023-10-03T17:40:25.495827+0000","objects_scrubbed":7,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:05:44.939663+0000","scrub_duration":0.0133078,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1007,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":51,"num_read_kb":24,"num_write":46,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"93'2683","reported_seq":6507,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-10-03T17:51:10.682015+0000","last_change":"2023-10-03T17:40:26.513902+0000","last_active":"2023-10-03T17:51:10.682015+0000","last_peered":"2023-10-03T17:51:10.682015+0000","last_clean":"2023-10-03T17:51:10.682015+0000","last_became_active":"2023-10-03T17:34:42.276459+0000","last_became_peered":"2023-10-03T17:34:42.276459+0000","last_unstale":"2023-10-03T17:51:10.682015+0000","last_undegraded":"2023-10-03T17:51:10.682015+0000","last_fullsized":"2023-10-03T17:51:10.682015+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'964","last_scrub_stamp":"2023-10-03T17:40:26.513729+0000","last_deep_scrub":"46'964","last_deep_scrub_stamp":"2023-10-03T17:40:26.513729+0000","last_clean_scrub_stamp":"2023-10-03T17:40:26.513729+0000","objects_scrubbed":10,"log_size":2683,"log_dups_size":0,"ondisk_log_size":2683,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:41:33.182751+0000","scrub_duration":0.018190280999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":283,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":9987,"num_read_kb":8356,"num_write":373,"num_write_kb":303,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1731,"num_omap_keys":80,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"93'120992","reported_seq":199055,"reported_epoch":93,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T17:51:13.231944+0000","last_change":"2023-10-03T17:51:08.194043+0000","last_active":"2023-10-03T17:51:13.231944+0000","last_peered":"2023-10-03T17:51:13.231944+0000","last_clean":"2023-10-03T17:51:13.231944+0000","last_became_active":"2023-10-03T17:34:48.310701+0000","last_became_peered":"2023-10-03T17:34:48.310701+0000","last_unstale":"2023-10-03T17:51:13.231944+0000","last_undegraded":"2023-10-03T17:51:13.231944+0000","last_fullsized":"2023-10-03T17:51:13.231944+0000","mapping_epoch":16,"log_start":"79'110960","ondisk_log_start":"79'110960","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'80695","last_scrub_stamp":"2023-10-03T17:46:13.982571+0000","last_deep_scrub":"66'80695","last_deep_scrub_stamp":"2023-10-03T17:46:13.982571+0000","last_clean_scrub_stamp":"2023-10-03T17:46:13.982571+0000","objects_scrubbed":3410,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":58,"last_scrub_duration":363,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:41:58.496895+0000","scrub_duration":362.26574615599998,"objects_trimmed":92,"snaptrim_duration":1.8281656580000001,"stat_sum":{"num_bytes":5674975232,"num_objects":9445,"num_object_clones":8125,"num_object_copies":18890,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9445,"num_whiteouts":0,"num_read":93381,"num_read_kb":352948,"num_write":94810,"num_write_kb":379240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"54","length":"1e"},{"start":"7c","length":"8"},{"start":"8a","length":"2"},{"start":"90","length":"a"},{"start":"a4","length":"a"},{"start":"b8","length":"a"},{"start":"cc","length":"a"},{"start":"e0","length":"2"}]},{"pgid":"2.2","version":"93'2779","reported_seq":5829,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-10-03T17:51:10.777710+0000","last_change":"2023-10-03T17:40:08.675119+0000","last_active":"2023-10-03T17:51:10.777710+0000","last_peered":"2023-10-03T17:51:10.777710+0000","last_clean":"2023-10-03T17:51:10.777710+0000","last_became_active":"2023-10-03T17:34:42.270625+0000","last_became_peered":"2023-10-03T17:34:42.270625+0000","last_unstale":"2023-10-03T17:51:10.777710+0000","last_undegraded":"2023-10-03T17:51:10.777710+0000","last_fullsized":"2023-10-03T17:51:10.777710+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'841","last_scrub_stamp":"2023-10-03T17:40:08.674910+0000","last_deep_scrub":"46'841","last_deep_scrub_stamp":"2023-10-03T17:40:08.674910+0000","last_clean_scrub_stamp":"2023-10-03T17:40:08.674910+0000","objects_scrubbed":11,"log_size":2779,"log_dups_size":0,"ondisk_log_size":2779,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T21:05:42.790614+0000","scrub_duration":0.039189555000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":807,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":9148,"num_read_kb":7575,"num_write":402,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2564,"num_omap_keys":60,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"93'107328","reported_seq":183127,"reported_epoch":93,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T17:51:13.367491+0000","last_change":"2023-10-03T17:43:28.494717+0000","last_active":"2023-10-03T17:51:13.367491+0000","last_peered":"2023-10-03T17:51:13.367491+0000","last_clean":"2023-10-03T17:51:13.367491+0000","last_became_active":"2023-10-03T17:34:48.310555+0000","last_became_peered":"2023-10-03T17:34:48.310555+0000","last_unstale":"2023-10-03T17:51:13.367491+0000","last_undegraded":"2023-10-03T17:51:13.367491+0000","last_fullsized":"2023-10-03T17:51:13.367491+0000","mapping_epoch":16,"log_start":"79'97320","ondisk_log_start":"79'97320","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_clean_scrub_stamp":"2023-10-03T17:34:47.281095+0000","objects_scrubbed":7318,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":140,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 464s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086289699999999995,"stat_sum":{"num_bytes":5338726400,"num_objects":13919,"num_object_clones":12677,"num_object_copies":27838,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13919,"num_whiteouts":0,"num_read":87862,"num_read_kb":331380,"num_write":88348,"num_write_kb":353392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"93'248","reported_seq":2523,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-10-03T17:51:09.903034+0000","last_change":"2023-10-03T17:40:17.641493+0000","last_active":"2023-10-03T17:51:09.903034+0000","last_peered":"2023-10-03T17:51:09.903034+0000","last_clean":"2023-10-03T17:51:09.903034+0000","last_became_active":"2023-10-03T17:34:42.278196+0000","last_became_peered":"2023-10-03T17:34:42.278196+0000","last_unstale":"2023-10-03T17:51:09.903034+0000","last_undegraded":"2023-10-03T17:51:09.903034+0000","last_fullsized":"2023-10-03T17:51:09.903034+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'30","last_scrub_stamp":"2023-10-03T17:40:17.640969+0000","last_deep_scrub":"46'30","last_deep_scrub_stamp":"2023-10-03T17:40:17.640969+0000","last_clean_scrub_stamp":"2023-10-03T17:40:17.640969+0000","objects_scrubbed":6,"log_size":248,"log_dups_size":0,"ondisk_log_size":248,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:04:02.954896+0000","scrub_duration":0.071054565,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":521,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":6814,"num_read_kb":5635,"num_write":272,"num_write_kb":218,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":903,"num_omap_keys":30,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"93'119888","reported_seq":194157,"reported_epoch":93,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T17:51:13.218750+0000","last_change":"2023-10-03T17:51:09.915425+0000","last_active":"2023-10-03T17:51:13.218750+0000","last_peered":"2023-10-03T17:51:13.218750+0000","last_clean":"2023-10-03T17:51:13.218750+0000","last_became_active":"2023-10-03T17:34:48.310912+0000","last_became_peered":"2023-10-03T17:34:48.310912+0000","last_unstale":"2023-10-03T17:51:13.218750+0000","last_undegraded":"2023-10-03T17:51:13.218750+0000","last_fullsized":"2023-10-03T17:51:13.218750+0000","mapping_epoch":16,"log_start":"79'109819","ondisk_log_start":"79'109819","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'58196","last_scrub_stamp":"2023-10-03T17:43:26.193075+0000","last_deep_scrub":"53'58196","last_deep_scrub_stamp":"2023-10-03T17:43:26.193075+0000","last_clean_scrub_stamp":"2023-10-03T17:43:26.193075+0000","objects_scrubbed":2326,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":23,"last_scrub_duration":196,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:10:30.229402+0000","scrub_duration":195.44739730800001,"objects_trimmed":99,"snaptrim_duration":3.5445642820000001,"stat_sum":{"num_bytes":5468741632,"num_objects":6982,"num_object_clones":5711,"num_object_copies":13964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6982,"num_whiteouts":0,"num_read":90049,"num_read_kb":341240,"num_write":91256,"num_write_kb":365024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"54","length":"1e"},{"start":"7c","length":"8"},{"start":"8a","length":"2"},{"start":"90","length":"a"},{"start":"a4","length":"a"},{"start":"b8","length":"a"},{"start":"cc","length":"a"},{"start":"e0","length":"a"},{"start":"f4","length":"a"},{"start":"108","length":"a"},{"start":"11c","length":"7"}]},{"pgid":"2.0","version":"91'128","reported_seq":989,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-10-03T17:51:06.386811+0000","last_change":"2023-10-03T17:40:10.708344+0000","last_active":"2023-10-03T17:51:06.386811+0000","last_peered":"2023-10-03T17:51:06.386811+0000","last_clean":"2023-10-03T17:51:06.386811+0000","last_became_active":"2023-10-03T17:34:42.272306+0000","last_became_peered":"2023-10-03T17:34:42.272306+0000","last_unstale":"2023-10-03T17:51:06.386811+0000","last_undegraded":"2023-10-03T17:51:06.386811+0000","last_fullsized":"2023-10-03T17:51:06.386811+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'20","last_scrub_stamp":"2023-10-03T17:40:10.708060+0000","last_deep_scrub":"46'20","last_deep_scrub_stamp":"2023-10-03T17:40:10.708060+0000","last_clean_scrub_stamp":"2023-10-03T17:40:10.708060+0000","objects_scrubbed":7,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:03:09.097252+0000","scrub_duration":0.042279491000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":905,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":2322,"num_read_kb":1905,"num_write":131,"num_write_kb":101,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":301,"num_omap_keys":10,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"93'120213","reported_seq":195625,"reported_epoch":93,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T17:51:13.405084+0000","last_change":"2023-10-03T17:51:09.692724+0000","last_active":"2023-10-03T17:51:13.405084+0000","last_peered":"2023-10-03T17:51:13.405084+0000","last_clean":"2023-10-03T17:51:13.405084+0000","last_became_active":"2023-10-03T17:34:48.310612+0000","last_became_peered":"2023-10-03T17:34:48.310612+0000","last_unstale":"2023-10-03T17:51:13.405084+0000","last_undegraded":"2023-10-03T17:51:13.405084+0000","last_fullsized":"2023-10-03T17:51:13.405084+0000","mapping_epoch":16,"log_start":"79'110203","ondisk_log_start":"79'110203","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'75807","last_scrub_stamp":"2023-10-03T17:45:43.833310+0000","last_deep_scrub":"61'75807","last_deep_scrub_stamp":"2023-10-03T17:45:43.833310+0000","last_clean_scrub_stamp":"2023-10-03T17:45:43.833310+0000","objects_scrubbed":3607,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":41,"last_scrub_duration":315,"scrub_schedule":"queued for deep scrub","scrub_duration":314.26374841099999,"objects_trimmed":115,"snaptrim_duration":3.3089948759999999,"stat_sum":{"num_bytes":5534441491,"num_objects":8192,"num_object_clones":6904,"num_object_copies":16384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8192,"num_whiteouts":0,"num_read":90832,"num_read_kb":343756,"num_write":92251,"num_write_kb":368998,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"54","length":"1e"},{"start":"7c","length":"8"},{"start":"8a","length":"2"},{"start":"90","length":"a"},{"start":"a4","length":"a"},{"start":"b8","length":"a"},{"start":"cc","length":"a"},{"start":"e0","length":"a"},{"start":"f4","length":"9"}]},{"pgid":"2.3","version":"91'49","reported_seq":185,"reported_epoch":92,"state":"active+clean","last_fresh":"2023-10-03T17:51:03.434333+0000","last_change":"2023-10-03T17:40:24.472734+0000","last_active":"2023-10-03T17:51:03.434333+0000","last_peered":"2023-10-03T17:51:03.434333+0000","last_clean":"2023-10-03T17:51:03.434333+0000","last_became_active":"2023-10-03T17:34:42.270194+0000","last_became_peered":"2023-10-03T17:34:42.270194+0000","last_unstale":"2023-10-03T17:51:03.434333+0000","last_undegraded":"2023-10-03T17:51:03.434333+0000","last_fullsized":"2023-10-03T17:51:03.434333+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'8","last_scrub_stamp":"2023-10-03T17:40:24.472573+0000","last_deep_scrub":"46'8","last_deep_scrub_stamp":"2023-10-03T17:40:24.472573+0000","last_clean_scrub_stamp":"2023-10-03T17:40:24.472573+0000","objects_scrubbed":4,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T18:28:04.735033+0000","scrub_duration":0.024990637999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":487,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":43,"num_read_kb":25,"num_write":45,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"46'99","reported_seq":238,"reported_epoch":92,"state":"active+clean","last_fresh":"2023-10-03T17:51:03.439004+0000","last_change":"2023-10-03T17:40:10.452341+0000","last_active":"2023-10-03T17:51:03.439004+0000","last_peered":"2023-10-03T17:51:03.439004+0000","last_clean":"2023-10-03T17:51:03.439004+0000","last_became_active":"2023-10-03T17:34:39.502393+0000","last_became_peered":"2023-10-03T17:34:39.502393+0000","last_unstale":"2023-10-03T17:51:03.439004+0000","last_undegraded":"2023-10-03T17:51:03.439004+0000","last_fullsized":"2023-10-03T17:51:03.439004+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'99","last_scrub_stamp":"2023-10-03T17:40:10.449099+0000","last_deep_scrub":"46'99","last_deep_scrub_stamp":"2023-10-03T17:40:10.449099+0000","last_clean_scrub_stamp":"2023-10-03T17:40:10.449099+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T20:18:04.958666+0000","scrub_duration":0.043225012,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":22016884755,"num_objects":38538,"num_object_clones":33417,"num_object_copies":77076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38538,"num_whiteouts":0,"num_read":362124,"num_read_kb":1369324,"num_write":366665,"num_write_kb":1466654,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3004198912,"data_stored":16681463846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40119,"ondisk_log_size":40119,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":5372,"num_objects":143,"num_object_clones":0,"num_object_copies":286,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":143,"num_whiteouts":0,"num_read":48749,"num_read_kb":40742,"num_write":2063,"num_write_kb":1643,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":983040,"data_stored":10744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7029,"internal_metadata":0},"log_size":11320,"ondisk_log_size":11320,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":7460,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738573,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2202360,"kb_used_data":1468016,"kb_used_omap":2891,"kb_used_meta":731444,"kb_avail":92169480,"statfs":{"total":96636764160,"available":94381547520,"internally_reserved":0,"allocated":1503248384,"data_stored":8345855067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2960945,"internal_metadata":748999119},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,3,7,19,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 17:51:06 2023","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.623,"15min":0.47799999999999998},"min":{"1min":0.17499999999999999,"5min":0.17199999999999999,"15min":0.14299999999999999},"max":{"1min":2.0310000000000001,"5min":14.787000000000001,"15min":14.787000000000001},"last":0.373},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.61599999999999999,"15min":0.47499999999999998},"min":{"1min":0.20899999999999999,"5min":0.17999999999999999,"15min":0.17100000000000001},"max":{"1min":0.745,"5min":14.919,"15min":14.919},"last":0.218}]},{"osd":1,"last update":"Tue Oct 3 17:51:06 2023","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.54600000000000004,"15min":0.52400000000000002},"min":{"1min":0.218,"5min":0.218,"15min":0.16600000000000001},"max":{"1min":4.7069999999999999,"5min":6.7539999999999996,"15min":8.1059999999999999},"last":0.23300000000000001},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.53200000000000003,"15min":0.46999999999999997},"min":{"1min":0.23699999999999999,"5min":0.223,"15min":0.17699999999999999},"max":{"1min":1.621,"5min":5.194,"15min":5.194},"last":0.55900000000000005}]}]},{"osd":1,"up_from":8,"seq":34359738571,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1984632,"kb_used_data":723632,"kb_used_omap":1,"kb_used_meta":1260990,"kb_avail":92387208,"statfs":{"total":96636764160,"available":94604500992,"internally_reserved":0,"allocated":740999168,"data_stored":4784845822,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":1291254214},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,1,5,9,4,7,7],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 17:50:17 2023","interfaces":[{"interface":"back","average":{"1min":0.33000000000000002,"5min":0.33500000000000002,"15min":0.35199999999999998},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.161},"max":{"1min":0.84899999999999998,"5min":2.7050000000000001,"15min":4.1879999999999997},"last":0.26700000000000002},{"interface":"front","average":{"1min":0.313,"5min":0.39100000000000001,"15min":0.39700000000000002},"min":{"1min":0.21199999999999999,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.53400000000000003,"5min":5.8959999999999999,"15min":5.8959999999999999},"last":0.19500000000000001}]},{"osd":2,"last update":"Tue Oct 3 17:50:17 2023","interfaces":[{"interface":"back","average":{"1min":0.39700000000000002,"5min":0.47999999999999998,"15min":0.41899999999999998},"min":{"1min":0.223,"5min":0.17599999999999999,"15min":0.159},"max":{"1min":1.391,"5min":4.5519999999999996,"15min":4.5519999999999996},"last":0.29099999999999998},{"interface":"front","average":{"1min":0.40699999999999997,"5min":0.42899999999999999,"15min":0.39300000000000002},"min":{"1min":0.27500000000000002,"5min":0.23200000000000001,"15min":0.16400000000000001},"max":{"1min":0.91100000000000003,"5min":2.6970000000000001,"15min":3.0699999999999998},"last":0.24299999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738571,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2063056,"kb_used_data":745096,"kb_used_omap":1,"kb_used_meta":1317950,"kb_avail":92308784,"statfs":{"total":96636764160,"available":94524194816,"internally_reserved":0,"allocated":762978304,"data_stored":3542039666,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":1349581254},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,4,0,0,1,3,8,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 17:51:03 2023","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.40799999999999997,"15min":0.40999999999999998},"min":{"1min":0.151,"5min":0.151,"15min":0.151},"max":{"1min":3.9430000000000001,"5min":3.9430000000000001,"15min":5.7649999999999997},"last":0.248},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.48599999999999999,"15min":0.47099999999999997},"min":{"1min":0.23000000000000001,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":3.9169999999999998,"5min":3.9169999999999998,"15min":9.4770000000000003},"last":0.26500000000000001}]},{"osd":2,"last update":"Tue Oct 3 17:51:03 2023","interfaces":[{"interface":"back","average":{"1min":0.33100000000000002,"5min":0.46300000000000002,"15min":0.497},"min":{"1min":0.17399999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.95599999999999996,"5min":3.4540000000000002,"15min":9.4149999999999991},"last":0.215},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.438,"15min":0.47399999999999998},"min":{"1min":0.217,"5min":0.217,"15min":0.189},"max":{"1min":1.046,"5min":3.4249999999999998,"15min":5.218},"last":0.23100000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":3459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":5372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":172032,"data_stored":1913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7029,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":762130432,"data_stored":3549749248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":739741696,"data_stored":4785848339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1502326784,"data_stored":8345866259,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-03T17:51:15.291 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-03T17:34:47.281095+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=34, tm_sec=47, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=40, tm_sec=6, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T17:51:15.292 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T17:51:30.008 INFO:tasks.ceph.mgr.x.smithi111.stderr:2023-10-03T17:51:30.006+0000 7fe48bb97640 -1 calc_pg_upmaps abort due to max <= 0 2023-10-03T17:51:35.292 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T17:51:35.977 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-03T17:51:35.977 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-10-03T17:51:36.043 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":600,"stamp":"2023-10-03T17:51:34.200219+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":22043973423,"num_objects":39594,"num_object_clones":34328,"num_object_copies":79188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39594,"num_whiteouts":0,"num_read":418186,"num_read_kb":1434072,"num_write":374444,"num_write_kb":1492507,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":51793,"ondisk_log_size":51793,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":6438560,"kb_used_data":2980872,"kb_used_omap":2894,"kb_used_meta":3454769,"kb_avail":276676960,"statfs":{"total":289910292480,"available":283317207040,"internally_reserved":0,"allocated":3052412928,"data_stored":17293976779,"data_compressed":7460,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":2964133,"internal_metadata":3537683803},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,2,2,15,14,16,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":20348928,"num_objects":483,"num_object_clones":483,"num_object_copies":966,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":483,"num_whiteouts":0,"num_read":5173,"num_read_kb":18024,"num_write":4257,"num_write_kb":17028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002010"},"pg_stats":[{"pgid":"2.7","version":"93'2054","reported_seq":4393,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-10-03T17:51:33.301845+0000","last_change":"2023-10-03T17:40:17.407011+0000","last_active":"2023-10-03T17:51:33.301845+0000","last_peered":"2023-10-03T17:51:33.301845+0000","last_clean":"2023-10-03T17:51:33.301845+0000","last_became_active":"2023-10-03T17:34:42.276624+0000","last_became_peered":"2023-10-03T17:34:42.276624+0000","last_unstale":"2023-10-03T17:51:33.301845+0000","last_undegraded":"2023-10-03T17:51:33.301845+0000","last_fullsized":"2023-10-03T17:51:33.301845+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'673","last_scrub_stamp":"2023-10-03T17:40:17.406813+0000","last_deep_scrub":"46'673","last_deep_scrub_stamp":"2023-10-03T17:40:17.406813+0000","last_clean_scrub_stamp":"2023-10-03T17:40:17.406813+0000","objects_scrubbed":7,"log_size":2054,"log_dups_size":0,"ondisk_log_size":2054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T21:38:35.056889+0000","scrub_duration":0.013778076,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":690,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":7043,"num_read_kb":5819,"num_write":298,"num_write_kb":233,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":903,"num_omap_keys":30,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"93'3398","reported_seq":8811,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-10-03T17:51:33.305893+0000","last_change":"2023-10-03T17:40:13.376519+0000","last_active":"2023-10-03T17:51:33.305893+0000","last_peered":"2023-10-03T17:51:33.305893+0000","last_clean":"2023-10-03T17:51:33.305893+0000","last_became_active":"2023-10-03T17:34:42.271201+0000","last_became_peered":"2023-10-03T17:34:42.271201+0000","last_unstale":"2023-10-03T17:51:33.305893+0000","last_undegraded":"2023-10-03T17:51:33.305893+0000","last_fullsized":"2023-10-03T17:51:33.305893+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'1129","last_scrub_stamp":"2023-10-03T17:40:13.376332+0000","last_deep_scrub":"46'1129","last_deep_scrub_stamp":"2023-10-03T17:40:13.376332+0000","last_clean_scrub_stamp":"2023-10-03T17:40:13.376332+0000","objects_scrubbed":11,"log_size":3398,"log_dups_size":0,"ondisk_log_size":3398,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T18:59:17.088585+0000","scrub_duration":0.015998675,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":13953,"num_read_kb":11918,"num_write":496,"num_write_kb":404,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2912,"num_omap_keys":93,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"91'53","reported_seq":192,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-10-03T17:51:06.375767+0000","last_change":"2023-10-03T17:40:25.496010+0000","last_active":"2023-10-03T17:51:06.375767+0000","last_peered":"2023-10-03T17:51:06.375767+0000","last_clean":"2023-10-03T17:51:06.375767+0000","last_became_active":"2023-10-03T17:34:42.276941+0000","last_became_peered":"2023-10-03T17:34:42.276941+0000","last_unstale":"2023-10-03T17:51:06.375767+0000","last_undegraded":"2023-10-03T17:51:06.375767+0000","last_fullsized":"2023-10-03T17:51:06.375767+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'14","last_scrub_stamp":"2023-10-03T17:40:25.495827+0000","last_deep_scrub":"46'14","last_deep_scrub_stamp":"2023-10-03T17:40:25.495827+0000","last_clean_scrub_stamp":"2023-10-03T17:40:25.495827+0000","objects_scrubbed":7,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:05:44.939663+0000","scrub_duration":0.0133078,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1007,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":51,"num_read_kb":24,"num_write":46,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"93'2719","reported_seq":6626,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-10-03T17:51:33.303423+0000","last_change":"2023-10-03T17:40:26.513902+0000","last_active":"2023-10-03T17:51:33.303423+0000","last_peered":"2023-10-03T17:51:33.303423+0000","last_clean":"2023-10-03T17:51:33.303423+0000","last_became_active":"2023-10-03T17:34:42.276459+0000","last_became_peered":"2023-10-03T17:34:42.276459+0000","last_unstale":"2023-10-03T17:51:33.303423+0000","last_undegraded":"2023-10-03T17:51:33.303423+0000","last_fullsized":"2023-10-03T17:51:33.303423+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'964","last_scrub_stamp":"2023-10-03T17:40:26.513729+0000","last_deep_scrub":"46'964","last_deep_scrub_stamp":"2023-10-03T17:40:26.513729+0000","last_clean_scrub_stamp":"2023-10-03T17:40:26.513729+0000","objects_scrubbed":10,"log_size":2719,"log_dups_size":0,"ondisk_log_size":2719,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:41:33.182751+0000","scrub_duration":0.018190280999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":283,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":10273,"num_read_kb":8593,"num_write":373,"num_write_kb":303,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1731,"num_omap_keys":80,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"93'123569","reported_seq":202543,"reported_epoch":93,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T17:51:33.496162+0000","last_change":"2023-10-03T17:51:29.517564+0000","last_active":"2023-10-03T17:51:33.496162+0000","last_peered":"2023-10-03T17:51:33.496162+0000","last_clean":"2023-10-03T17:51:33.496162+0000","last_became_active":"2023-10-03T17:34:48.310701+0000","last_became_peered":"2023-10-03T17:34:48.310701+0000","last_unstale":"2023-10-03T17:51:33.496162+0000","last_undegraded":"2023-10-03T17:51:33.496162+0000","last_fullsized":"2023-10-03T17:51:33.496162+0000","mapping_epoch":16,"log_start":"88'113472","ondisk_log_start":"88'113472","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'80695","last_scrub_stamp":"2023-10-03T17:46:13.982571+0000","last_deep_scrub":"66'80695","last_deep_scrub_stamp":"2023-10-03T17:46:13.982571+0000","last_clean_scrub_stamp":"2023-10-03T17:46:13.982571+0000","objects_scrubbed":3410,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":53,"last_scrub_duration":363,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:41:58.496895+0000","scrub_duration":362.26574615599998,"objects_trimmed":414,"snaptrim_duration":23.143643546,"stat_sum":{"num_bytes":5681692672,"num_objects":9629,"num_object_clones":8309,"num_object_copies":19258,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9629,"num_whiteouts":0,"num_read":94935,"num_read_kb":359052,"num_write":96237,"num_write_kb":384948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"54","length":"1e"},{"start":"7c","length":"8"},{"start":"8a","length":"2"},{"start":"90","length":"a"},{"start":"a4","length":"a"},{"start":"b8","length":"a"},{"start":"cc","length":"a"},{"start":"e0","length":"7"}]},{"pgid":"2.2","version":"93'2807","reported_seq":5920,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-10-03T17:51:29.846227+0000","last_change":"2023-10-03T17:40:08.675119+0000","last_active":"2023-10-03T17:51:29.846227+0000","last_peered":"2023-10-03T17:51:29.846227+0000","last_clean":"2023-10-03T17:51:29.846227+0000","last_became_active":"2023-10-03T17:34:42.270625+0000","last_became_peered":"2023-10-03T17:34:42.270625+0000","last_unstale":"2023-10-03T17:51:29.846227+0000","last_undegraded":"2023-10-03T17:51:29.846227+0000","last_fullsized":"2023-10-03T17:51:29.846227+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'841","last_scrub_stamp":"2023-10-03T17:40:08.674910+0000","last_deep_scrub":"46'841","last_deep_scrub_stamp":"2023-10-03T17:40:08.674910+0000","last_clean_scrub_stamp":"2023-10-03T17:40:08.674910+0000","objects_scrubbed":11,"log_size":2807,"log_dups_size":0,"ondisk_log_size":2807,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T21:05:42.790614+0000","scrub_duration":0.039189555000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":807,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":9414,"num_read_kb":7794,"num_write":402,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2564,"num_omap_keys":60,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"93'109451","reported_seq":186160,"reported_epoch":93,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T17:51:33.623321+0000","last_change":"2023-10-03T17:43:28.494717+0000","last_active":"2023-10-03T17:51:33.623321+0000","last_peered":"2023-10-03T17:51:33.623321+0000","last_clean":"2023-10-03T17:51:33.623321+0000","last_became_active":"2023-10-03T17:34:48.310555+0000","last_became_peered":"2023-10-03T17:34:48.310555+0000","last_unstale":"2023-10-03T17:51:33.623321+0000","last_undegraded":"2023-10-03T17:51:33.623321+0000","last_fullsized":"2023-10-03T17:51:33.623321+0000","mapping_epoch":16,"log_start":"79'99425","ondisk_log_start":"79'99425","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_clean_scrub_stamp":"2023-10-03T17:34:47.281095+0000","objects_scrubbed":7653,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":140,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 485s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086289699999999995,"stat_sum":{"num_bytes":5345079296,"num_objects":14408,"num_object_clones":13166,"num_object_copies":28816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14408,"num_whiteouts":0,"num_read":89250,"num_read_kb":336820,"num_write":89675,"num_write_kb":358700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"93'248","reported_seq":2568,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-10-03T17:51:29.847613+0000","last_change":"2023-10-03T17:40:17.641493+0000","last_active":"2023-10-03T17:51:29.847613+0000","last_peered":"2023-10-03T17:51:29.847613+0000","last_clean":"2023-10-03T17:51:29.847613+0000","last_became_active":"2023-10-03T17:34:42.278196+0000","last_became_peered":"2023-10-03T17:34:42.278196+0000","last_unstale":"2023-10-03T17:51:29.847613+0000","last_undegraded":"2023-10-03T17:51:29.847613+0000","last_fullsized":"2023-10-03T17:51:29.847613+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'30","last_scrub_stamp":"2023-10-03T17:40:17.640969+0000","last_deep_scrub":"46'30","last_deep_scrub_stamp":"2023-10-03T17:40:17.640969+0000","last_clean_scrub_stamp":"2023-10-03T17:40:17.640969+0000","objects_scrubbed":6,"log_size":248,"log_dups_size":0,"ondisk_log_size":248,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:04:02.954896+0000","scrub_duration":0.071054565,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":521,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":7012,"num_read_kb":5797,"num_write":272,"num_write_kb":218,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":903,"num_omap_keys":30,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"93'122467","reported_seq":197522,"reported_epoch":93,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T17:51:33.428929+0000","last_change":"2023-10-03T17:51:33.354459+0000","last_active":"2023-10-03T17:51:33.428929+0000","last_peered":"2023-10-03T17:51:33.428929+0000","last_clean":"2023-10-03T17:51:33.428929+0000","last_became_active":"2023-10-03T17:34:48.310912+0000","last_became_peered":"2023-10-03T17:34:48.310912+0000","last_unstale":"2023-10-03T17:51:33.428929+0000","last_undegraded":"2023-10-03T17:51:33.428929+0000","last_fullsized":"2023-10-03T17:51:33.428929+0000","mapping_epoch":16,"log_start":"88'112435","ondisk_log_start":"88'112435","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'58196","last_scrub_stamp":"2023-10-03T17:43:26.193075+0000","last_deep_scrub":"53'58196","last_deep_scrub_stamp":"2023-10-03T17:43:26.193075+0000","last_clean_scrub_stamp":"2023-10-03T17:43:26.193075+0000","objects_scrubbed":2326,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":17,"last_scrub_duration":196,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:10:30.229402+0000","scrub_duration":195.44739730800001,"objects_trimmed":438,"snaptrim_duration":26.983972940000001,"stat_sum":{"num_bytes":5475794944,"num_objects":7118,"num_object_clones":5847,"num_object_copies":14236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7118,"num_whiteouts":0,"num_read":91447,"num_read_kb":346752,"num_write":92682,"num_write_kb":370728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"54","length":"1e"},{"start":"7c","length":"8"},{"start":"8a","length":"2"},{"start":"90","length":"a"},{"start":"a4","length":"a"},{"start":"b8","length":"a"},{"start":"cc","length":"a"},{"start":"e0","length":"a"},{"start":"f4","length":"a"},{"start":"108","length":"a"},{"start":"11c","length":"a"},{"start":"130","length":"3"}]},{"pgid":"2.0","version":"91'128","reported_seq":1004,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-10-03T17:51:25.164372+0000","last_change":"2023-10-03T17:40:10.708344+0000","last_active":"2023-10-03T17:51:25.164372+0000","last_peered":"2023-10-03T17:51:25.164372+0000","last_clean":"2023-10-03T17:51:25.164372+0000","last_became_active":"2023-10-03T17:34:42.272306+0000","last_became_peered":"2023-10-03T17:34:42.272306+0000","last_unstale":"2023-10-03T17:51:25.164372+0000","last_undegraded":"2023-10-03T17:51:25.164372+0000","last_fullsized":"2023-10-03T17:51:25.164372+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'20","last_scrub_stamp":"2023-10-03T17:40:10.708060+0000","last_deep_scrub":"46'20","last_deep_scrub_stamp":"2023-10-03T17:40:10.708060+0000","last_clean_scrub_stamp":"2023-10-03T17:40:10.708060+0000","objects_scrubbed":7,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:03:09.097252+0000","scrub_duration":0.042279491000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":905,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":2388,"num_read_kb":1959,"num_write":131,"num_write_kb":101,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":301,"num_omap_keys":10,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"93'122800","reported_seq":199034,"reported_epoch":93,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T17:51:33.631254+0000","last_change":"2023-10-03T17:51:33.304373+0000","last_active":"2023-10-03T17:51:33.631254+0000","last_peered":"2023-10-03T17:51:33.631254+0000","last_clean":"2023-10-03T17:51:33.631254+0000","last_became_active":"2023-10-03T17:34:48.310612+0000","last_became_peered":"2023-10-03T17:34:48.310612+0000","last_unstale":"2023-10-03T17:51:33.631254+0000","last_undegraded":"2023-10-03T17:51:33.631254+0000","last_fullsized":"2023-10-03T17:51:33.631254+0000","mapping_epoch":16,"log_start":"88'112717","ondisk_log_start":"88'112717","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'75807","last_scrub_stamp":"2023-10-03T17:45:43.833310+0000","last_deep_scrub":"61'75807","last_deep_scrub_stamp":"2023-10-03T17:45:43.833310+0000","last_clean_scrub_stamp":"2023-10-03T17:45:43.833310+0000","objects_scrubbed":3607,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":34,"last_scrub_duration":315,"scrub_schedule":"queued for deep scrub","scrub_duration":314.26374841099999,"objects_trimmed":477,"snaptrim_duration":26.920979823,"stat_sum":{"num_bytes":5540810771,"num_objects":8294,"num_object_clones":7006,"num_object_copies":16588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8294,"num_whiteouts":0,"num_read":92271,"num_read_kb":349428,"num_write":93650,"num_write_kb":374594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"54","length":"1e"},{"start":"7c","length":"8"},{"start":"8a","length":"2"},{"start":"90","length":"a"},{"start":"a4","length":"a"},{"start":"b8","length":"a"},{"start":"cc","length":"a"},{"start":"e0","length":"a"},{"start":"f4","length":"a"},{"start":"108","length":"6"}]},{"pgid":"2.3","version":"91'49","reported_seq":186,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-10-03T17:51:06.375691+0000","last_change":"2023-10-03T17:40:24.472734+0000","last_active":"2023-10-03T17:51:06.375691+0000","last_peered":"2023-10-03T17:51:06.375691+0000","last_clean":"2023-10-03T17:51:06.375691+0000","last_became_active":"2023-10-03T17:34:42.270194+0000","last_became_peered":"2023-10-03T17:34:42.270194+0000","last_unstale":"2023-10-03T17:51:06.375691+0000","last_undegraded":"2023-10-03T17:51:06.375691+0000","last_fullsized":"2023-10-03T17:51:06.375691+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'8","last_scrub_stamp":"2023-10-03T17:40:24.472573+0000","last_deep_scrub":"46'8","last_deep_scrub_stamp":"2023-10-03T17:40:24.472573+0000","last_clean_scrub_stamp":"2023-10-03T17:40:24.472573+0000","objects_scrubbed":4,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T18:28:04.735033+0000","scrub_duration":0.024990637999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":487,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":43,"num_read_kb":25,"num_write":45,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"46'99","reported_seq":238,"reported_epoch":92,"state":"active+clean","last_fresh":"2023-10-03T17:51:03.439004+0000","last_change":"2023-10-03T17:40:10.452341+0000","last_active":"2023-10-03T17:51:03.439004+0000","last_peered":"2023-10-03T17:51:03.439004+0000","last_clean":"2023-10-03T17:51:03.439004+0000","last_became_active":"2023-10-03T17:34:39.502393+0000","last_became_peered":"2023-10-03T17:34:39.502393+0000","last_unstale":"2023-10-03T17:51:03.439004+0000","last_undegraded":"2023-10-03T17:51:03.439004+0000","last_fullsized":"2023-10-03T17:51:03.439004+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'99","last_scrub_stamp":"2023-10-03T17:40:10.449099+0000","last_deep_scrub":"46'99","last_deep_scrub_stamp":"2023-10-03T17:40:10.449099+0000","last_clean_scrub_stamp":"2023-10-03T17:40:10.449099+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T20:18:04.958666+0000","scrub_duration":0.043225012,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":22043377683,"num_objects":39449,"num_object_clones":34328,"num_object_copies":78898,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39449,"num_whiteouts":0,"num_read":367903,"num_read_kb":1392052,"num_write":372244,"num_write_kb":1488970,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3049938944,"data_stored":17293033510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40238,"ondisk_log_size":40238,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":5372,"num_objects":143,"num_object_clones":0,"num_object_copies":286,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":143,"num_whiteouts":0,"num_read":50177,"num_read_kb":41929,"num_write":2063,"num_write_kb":1643,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":983040,"data_stored":10744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7029,"internal_metadata":0},"log_size":11456,"ondisk_log_size":11456,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":7460,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738577,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2300784,"kb_used_data":1490088,"kb_used_omap":2891,"kb_used_meta":807796,"kb_avail":92071056,"statfs":{"total":96636764160,"available":94280761344,"internally_reserved":0,"allocated":1525850112,"data_stored":8646149211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2960945,"internal_metadata":827183567},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,2,1,4,7,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 17:51:06 2023","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.623,"15min":0.47799999999999998},"min":{"1min":0.17499999999999999,"5min":0.17199999999999999,"15min":0.14299999999999999},"max":{"1min":2.0310000000000001,"5min":14.787000000000001,"15min":14.787000000000001},"last":0.313},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.61599999999999999,"15min":0.47499999999999998},"min":{"1min":0.20899999999999999,"5min":0.17999999999999999,"15min":0.17100000000000001},"max":{"1min":0.745,"5min":14.919,"15min":14.919},"last":0.33100000000000002}]},{"osd":1,"last update":"Tue Oct 3 17:51:06 2023","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.54600000000000004,"15min":0.52400000000000002},"min":{"1min":0.218,"5min":0.218,"15min":0.16600000000000001},"max":{"1min":4.7069999999999999,"5min":6.7539999999999996,"15min":8.1059999999999999},"last":3.5659999999999998},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.53200000000000003,"15min":0.46999999999999997},"min":{"1min":0.23699999999999999,"5min":0.223,"15min":0.17699999999999999},"max":{"1min":1.621,"5min":5.194,"15min":5.194},"last":0.27800000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738575,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2031528,"kb_used_data":734432,"kb_used_omap":1,"kb_used_meta":1297086,"kb_avail":92340312,"statfs":{"total":96636764160,"available":94556479488,"internally_reserved":0,"allocated":752058368,"data_stored":4974900222,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":1328216518},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,0,3,6,5,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 17:51:17 2023","interfaces":[{"interface":"back","average":{"1min":0.317,"5min":0.33200000000000002,"15min":0.33800000000000002},"min":{"1min":0.184,"5min":0.17899999999999999,"15min":0.161},"max":{"1min":0.57799999999999996,"5min":2.7050000000000001,"15min":4.1879999999999997},"last":0.20100000000000001},{"interface":"front","average":{"1min":0.39600000000000002,"5min":0.371,"15min":0.38800000000000001},"min":{"1min":0.19500000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":2.2040000000000002,"5min":5.8959999999999999,"15min":5.8959999999999999},"last":0.23300000000000001}]},{"osd":2,"last update":"Tue Oct 3 17:51:17 2023","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.46300000000000002,"15min":0.40500000000000003},"min":{"1min":0.20200000000000001,"5min":0.17599999999999999,"15min":0.159},"max":{"1min":0.63700000000000001,"5min":4.5519999999999996,"15min":4.5519999999999996},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.34899999999999998,"5min":0.41499999999999998,"15min":0.378},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.16400000000000001},"max":{"1min":0.56399999999999995,"5min":2.6970000000000001,"15min":3.0699999999999998},"last":0.33800000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738575,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2106248,"kb_used_data":756352,"kb_used_omap":1,"kb_used_meta":1349886,"kb_avail":92265592,"statfs":{"total":96636764160,"available":94479966208,"internally_reserved":0,"allocated":774504448,"data_stored":3672927346,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":1382283718},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,8,1,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 17:51:03 2023","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.40799999999999997,"15min":0.40999999999999998},"min":{"1min":0.151,"5min":0.151,"15min":0.151},"max":{"1min":3.9430000000000001,"5min":3.9430000000000001,"15min":5.7649999999999997},"last":2.8340000000000001},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.48599999999999999,"15min":0.47099999999999997},"min":{"1min":0.23000000000000001,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":3.9169999999999998,"5min":3.9169999999999998,"15min":9.4770000000000003},"last":3.6930000000000001}]},{"osd":2,"last update":"Tue Oct 3 17:51:03 2023","interfaces":[{"interface":"back","average":{"1min":0.33100000000000002,"5min":0.46300000000000002,"15min":0.497},"min":{"1min":0.17399999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.95599999999999996,"5min":3.4540000000000002,"15min":9.4149999999999991},"last":0.25700000000000001},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.438,"15min":0.47399999999999998},"min":{"1min":0.217,"5min":0.217,"15min":0.189},"max":{"1min":1.046,"5min":3.4249999999999998,"15min":5.218},"last":0.28000000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":3459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":5372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":172032,"data_stored":1913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7029,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":773832704,"data_stored":3672576000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":750899200,"data_stored":4974571539,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1525207040,"data_stored":8645885971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-03T17:51:36.045 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-03T17:34:47.281095+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=34, tm_sec=47, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=40, tm_sec=6, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T17:51:36.045 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T17:51:56.046 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T17:51:56.851 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-03T17:51:56.852 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-10-03T17:51:56.895 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":611,"stamp":"2023-10-03T17:51:56.205494+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":22073018159,"num_objects":39169,"num_object_clones":33903,"num_object_copies":78338,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39169,"num_whiteouts":0,"num_read":424463,"num_read_kb":1458680,"num_write":380658,"num_write_kb":1517363,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":51919,"ondisk_log_size":51919,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":6695768,"kb_used_data":3030400,"kb_used_omap":2894,"kb_used_meta":3662449,"kb_avail":276419752,"statfs":{"total":289910292480,"available":283053826048,"internally_reserved":0,"allocated":3103129600,"data_stored":17173550283,"data_compressed":7460,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":2964133,"internal_metadata":3750348123},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,4,7,15,10,15,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":14569472,"num_objects":-330,"num_object_clones":-330,"num_object_copies":-660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-330,"num_whiteouts":0,"num_read":3146,"num_read_kb":12324,"num_write":3149,"num_write_kb":12596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002816"},"pg_stats":[{"pgid":"2.7","version":"93'2075","reported_seq":4426,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-10-03T17:51:51.905938+0000","last_change":"2023-10-03T17:40:17.407011+0000","last_active":"2023-10-03T17:51:51.905938+0000","last_peered":"2023-10-03T17:51:51.905938+0000","last_clean":"2023-10-03T17:51:51.905938+0000","last_became_active":"2023-10-03T17:34:42.276624+0000","last_became_peered":"2023-10-03T17:34:42.276624+0000","last_unstale":"2023-10-03T17:51:51.905938+0000","last_undegraded":"2023-10-03T17:51:51.905938+0000","last_fullsized":"2023-10-03T17:51:51.905938+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'673","last_scrub_stamp":"2023-10-03T17:40:17.406813+0000","last_deep_scrub":"46'673","last_deep_scrub_stamp":"2023-10-03T17:40:17.406813+0000","last_clean_scrub_stamp":"2023-10-03T17:40:17.406813+0000","objects_scrubbed":7,"log_size":2075,"log_dups_size":0,"ondisk_log_size":2075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T21:38:35.056889+0000","scrub_duration":0.013778076,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":690,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":7043,"num_read_kb":5819,"num_write":298,"num_write_kb":233,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":903,"num_omap_keys":30,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"93'3433","reported_seq":8867,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-10-03T17:51:51.923044+0000","last_change":"2023-10-03T17:40:13.376519+0000","last_active":"2023-10-03T17:51:51.923044+0000","last_peered":"2023-10-03T17:51:51.923044+0000","last_clean":"2023-10-03T17:51:51.923044+0000","last_became_active":"2023-10-03T17:34:42.271201+0000","last_became_peered":"2023-10-03T17:34:42.271201+0000","last_unstale":"2023-10-03T17:51:51.923044+0000","last_undegraded":"2023-10-03T17:51:51.923044+0000","last_fullsized":"2023-10-03T17:51:51.923044+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'1129","last_scrub_stamp":"2023-10-03T17:40:13.376332+0000","last_deep_scrub":"46'1129","last_deep_scrub_stamp":"2023-10-03T17:40:13.376332+0000","last_clean_scrub_stamp":"2023-10-03T17:40:13.376332+0000","objects_scrubbed":11,"log_size":3433,"log_dups_size":0,"ondisk_log_size":3433,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T18:59:17.088585+0000","scrub_duration":0.015998675,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":13953,"num_read_kb":11918,"num_write":496,"num_write_kb":404,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2912,"num_omap_keys":93,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"91'53","reported_seq":192,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-10-03T17:51:06.375767+0000","last_change":"2023-10-03T17:40:25.496010+0000","last_active":"2023-10-03T17:51:06.375767+0000","last_peered":"2023-10-03T17:51:06.375767+0000","last_clean":"2023-10-03T17:51:06.375767+0000","last_became_active":"2023-10-03T17:34:42.276941+0000","last_became_peered":"2023-10-03T17:34:42.276941+0000","last_unstale":"2023-10-03T17:51:06.375767+0000","last_undegraded":"2023-10-03T17:51:06.375767+0000","last_fullsized":"2023-10-03T17:51:06.375767+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'14","last_scrub_stamp":"2023-10-03T17:40:25.495827+0000","last_deep_scrub":"46'14","last_deep_scrub_stamp":"2023-10-03T17:40:25.495827+0000","last_clean_scrub_stamp":"2023-10-03T17:40:25.495827+0000","objects_scrubbed":7,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:05:44.939663+0000","scrub_duration":0.0133078,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1007,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":51,"num_read_kb":24,"num_write":46,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"93'2747","reported_seq":6670,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-10-03T17:51:51.921049+0000","last_change":"2023-10-03T17:40:26.513902+0000","last_active":"2023-10-03T17:51:51.921049+0000","last_peered":"2023-10-03T17:51:51.921049+0000","last_clean":"2023-10-03T17:51:51.921049+0000","last_became_active":"2023-10-03T17:34:42.276459+0000","last_became_peered":"2023-10-03T17:34:42.276459+0000","last_unstale":"2023-10-03T17:51:51.921049+0000","last_undegraded":"2023-10-03T17:51:51.921049+0000","last_fullsized":"2023-10-03T17:51:51.921049+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'964","last_scrub_stamp":"2023-10-03T17:40:26.513729+0000","last_deep_scrub":"46'964","last_deep_scrub_stamp":"2023-10-03T17:40:26.513729+0000","last_clean_scrub_stamp":"2023-10-03T17:40:26.513729+0000","objects_scrubbed":10,"log_size":2747,"log_dups_size":0,"ondisk_log_size":2747,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:41:33.182751+0000","scrub_duration":0.018190280999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":283,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":10273,"num_read_kb":8593,"num_write":373,"num_write_kb":303,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1731,"num_omap_keys":80,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"93'126161","reported_seq":206404,"reported_epoch":93,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T17:51:53.694464+0000","last_change":"2023-10-03T17:51:50.594738+0000","last_active":"2023-10-03T17:51:53.694464+0000","last_peered":"2023-10-03T17:51:53.694464+0000","last_clean":"2023-10-03T17:51:53.694464+0000","last_became_active":"2023-10-03T17:34:48.310701+0000","last_became_peered":"2023-10-03T17:34:48.310701+0000","last_unstale":"2023-10-03T17:51:53.694464+0000","last_undegraded":"2023-10-03T17:51:53.694464+0000","last_fullsized":"2023-10-03T17:51:53.694464+0000","mapping_epoch":16,"log_start":"88'116085","ondisk_log_start":"88'116085","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'80695","last_scrub_stamp":"2023-10-03T17:46:13.982571+0000","last_deep_scrub":"66'80695","last_deep_scrub_stamp":"2023-10-03T17:46:13.982571+0000","last_clean_scrub_stamp":"2023-10-03T17:46:13.982571+0000","objects_scrubbed":3410,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":47,"last_scrub_duration":363,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:41:58.496895+0000","scrub_duration":362.26574615599998,"objects_trimmed":785,"snaptrim_duration":44.228537664999998,"stat_sum":{"num_bytes":5689462784,"num_objects":9456,"num_object_clones":8136,"num_object_copies":18912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9456,"num_whiteouts":0,"num_read":96560,"num_read_kb":365432,"num_write":97889,"num_write_kb":391556,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"54","length":"1e"},{"start":"7c","length":"8"},{"start":"8a","length":"2"},{"start":"90","length":"a"},{"start":"a4","length":"a"},{"start":"b8","length":"a"},{"start":"cc","length":"a"},{"start":"e0","length":"a"},{"start":"f4","length":"3"}]},{"pgid":"2.2","version":"93'2847","reported_seq":5976,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-10-03T17:51:50.331003+0000","last_change":"2023-10-03T17:40:08.675119+0000","last_active":"2023-10-03T17:51:50.331003+0000","last_peered":"2023-10-03T17:51:50.331003+0000","last_clean":"2023-10-03T17:51:50.331003+0000","last_became_active":"2023-10-03T17:34:42.270625+0000","last_became_peered":"2023-10-03T17:34:42.270625+0000","last_unstale":"2023-10-03T17:51:50.331003+0000","last_undegraded":"2023-10-03T17:51:50.331003+0000","last_fullsized":"2023-10-03T17:51:50.331003+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'841","last_scrub_stamp":"2023-10-03T17:40:08.674910+0000","last_deep_scrub":"46'841","last_deep_scrub_stamp":"2023-10-03T17:40:08.674910+0000","last_clean_scrub_stamp":"2023-10-03T17:40:08.674910+0000","objects_scrubbed":11,"log_size":2847,"log_dups_size":0,"ondisk_log_size":2847,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T21:05:42.790614+0000","scrub_duration":0.039189555000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":807,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":9414,"num_read_kb":7794,"num_write":402,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2564,"num_omap_keys":60,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"93'111376","reported_seq":189444,"reported_epoch":93,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T17:51:53.802495+0000","last_change":"2023-10-03T17:43:28.494717+0000","last_active":"2023-10-03T17:51:53.802495+0000","last_peered":"2023-10-03T17:51:53.802495+0000","last_clean":"2023-10-03T17:51:53.802495+0000","last_became_active":"2023-10-03T17:34:48.310555+0000","last_became_peered":"2023-10-03T17:34:48.310555+0000","last_unstale":"2023-10-03T17:51:53.802495+0000","last_undegraded":"2023-10-03T17:51:53.802495+0000","last_fullsized":"2023-10-03T17:51:53.802495+0000","mapping_epoch":16,"log_start":"88'101329","ondisk_log_start":"88'101329","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_clean_scrub_stamp":"2023-10-03T17:34:47.281095+0000","objects_scrubbed":7961,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":140,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 505s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086289699999999995,"stat_sum":{"num_bytes":5351682048,"num_objects":14593,"num_object_clones":13351,"num_object_copies":29186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14593,"num_whiteouts":0,"num_read":90771,"num_read_kb":342776,"num_write":91128,"num_write_kb":364512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"93'248","reported_seq":2580,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-10-03T17:51:50.353870+0000","last_change":"2023-10-03T17:40:17.641493+0000","last_active":"2023-10-03T17:51:50.353870+0000","last_peered":"2023-10-03T17:51:50.353870+0000","last_clean":"2023-10-03T17:51:50.353870+0000","last_became_active":"2023-10-03T17:34:42.278196+0000","last_became_peered":"2023-10-03T17:34:42.278196+0000","last_unstale":"2023-10-03T17:51:50.353870+0000","last_undegraded":"2023-10-03T17:51:50.353870+0000","last_fullsized":"2023-10-03T17:51:50.353870+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'30","last_scrub_stamp":"2023-10-03T17:40:17.640969+0000","last_deep_scrub":"46'30","last_deep_scrub_stamp":"2023-10-03T17:40:17.640969+0000","last_clean_scrub_stamp":"2023-10-03T17:40:17.640969+0000","objects_scrubbed":6,"log_size":248,"log_dups_size":0,"ondisk_log_size":248,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:04:02.954896+0000","scrub_duration":0.071054565,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":521,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":7012,"num_read_kb":5797,"num_write":272,"num_write_kb":218,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":903,"num_omap_keys":30,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"93'124976","reported_seq":201231,"reported_epoch":93,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T17:51:53.693602+0000","last_change":"2023-10-03T17:51:52.881376+0000","last_active":"2023-10-03T17:51:53.693602+0000","last_peered":"2023-10-03T17:51:53.693602+0000","last_clean":"2023-10-03T17:51:53.693602+0000","last_became_active":"2023-10-03T17:34:48.310912+0000","last_became_peered":"2023-10-03T17:34:48.310912+0000","last_unstale":"2023-10-03T17:51:53.693602+0000","last_undegraded":"2023-10-03T17:51:53.693602+0000","last_fullsized":"2023-10-03T17:51:53.693602+0000","mapping_epoch":16,"log_start":"88'114944","ondisk_log_start":"88'114944","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'58196","last_scrub_stamp":"2023-10-03T17:43:26.193075+0000","last_deep_scrub":"53'58196","last_deep_scrub_stamp":"2023-10-03T17:43:26.193075+0000","last_clean_scrub_stamp":"2023-10-03T17:43:26.193075+0000","objects_scrubbed":2326,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":10,"last_scrub_duration":196,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:10:30.229402+0000","scrub_duration":195.44739730800001,"objects_trimmed":865,"snaptrim_duration":46.510700319000001,"stat_sum":{"num_bytes":5482655744,"num_objects":6851,"num_object_clones":5580,"num_object_copies":13702,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6851,"num_whiteouts":0,"num_read":92987,"num_read_kb":352796,"num_write":94175,"num_write_kb":376700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"54","length":"1e"},{"start":"7c","length":"8"},{"start":"8a","length":"2"},{"start":"90","length":"a"},{"start":"a4","length":"a"},{"start":"b8","length":"a"},{"start":"cc","length":"a"},{"start":"e0","length":"a"},{"start":"f4","length":"a"},{"start":"108","length":"a"},{"start":"11c","length":"a"},{"start":"130","length":"a"}]},{"pgid":"2.0","version":"91'128","reported_seq":1008,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-10-03T17:51:45.517865+0000","last_change":"2023-10-03T17:40:10.708344+0000","last_active":"2023-10-03T17:51:45.517865+0000","last_peered":"2023-10-03T17:51:45.517865+0000","last_clean":"2023-10-03T17:51:45.517865+0000","last_became_active":"2023-10-03T17:34:42.272306+0000","last_became_peered":"2023-10-03T17:34:42.272306+0000","last_unstale":"2023-10-03T17:51:45.517865+0000","last_undegraded":"2023-10-03T17:51:45.517865+0000","last_fullsized":"2023-10-03T17:51:45.517865+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'20","last_scrub_stamp":"2023-10-03T17:40:10.708060+0000","last_deep_scrub":"46'20","last_deep_scrub_stamp":"2023-10-03T17:40:10.708060+0000","last_clean_scrub_stamp":"2023-10-03T17:40:10.708060+0000","objects_scrubbed":7,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:03:09.097252+0000","scrub_duration":0.042279491000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":905,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":2388,"num_read_kb":1959,"num_write":131,"num_write_kb":101,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":301,"num_omap_keys":10,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"93'125308","reported_seq":202791,"reported_epoch":93,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T17:51:53.813877+0000","last_change":"2023-10-03T17:51:50.891262+0000","last_active":"2023-10-03T17:51:53.813877+0000","last_peered":"2023-10-03T17:51:53.813877+0000","last_clean":"2023-10-03T17:51:53.813877+0000","last_became_active":"2023-10-03T17:34:48.310612+0000","last_became_peered":"2023-10-03T17:34:48.310612+0000","last_unstale":"2023-10-03T17:51:53.813877+0000","last_undegraded":"2023-10-03T17:51:53.813877+0000","last_fullsized":"2023-10-03T17:51:53.813877+0000","mapping_epoch":16,"log_start":"88'115223","ondisk_log_start":"88'115223","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'75807","last_scrub_stamp":"2023-10-03T17:45:43.833310+0000","last_deep_scrub":"61'75807","last_deep_scrub_stamp":"2023-10-03T17:45:43.833310+0000","last_clean_scrub_stamp":"2023-10-03T17:45:43.833310+0000","objects_scrubbed":3607,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":29,"last_scrub_duration":315,"scrub_schedule":"queued for deep scrub","scrub_duration":314.26374841099999,"objects_trimmed":831,"snaptrim_duration":44.507864724999997,"stat_sum":{"num_bytes":5548621843,"num_objects":8124,"num_object_clones":6836,"num_object_copies":16248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8124,"num_whiteouts":0,"num_read":93862,"num_read_kb":355656,"num_write":95266,"num_write_kb":381058,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"54","length":"1e"},{"start":"7c","length":"8"},{"start":"8a","length":"2"},{"start":"90","length":"a"},{"start":"a4","length":"a"},{"start":"b8","length":"a"},{"start":"cc","length":"a"},{"start":"e0","length":"a"},{"start":"f4","length":"a"},{"start":"108","length":"a"},{"start":"11c","length":"1"}]},{"pgid":"2.3","version":"91'49","reported_seq":186,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-10-03T17:51:06.375691+0000","last_change":"2023-10-03T17:40:24.472734+0000","last_active":"2023-10-03T17:51:06.375691+0000","last_peered":"2023-10-03T17:51:06.375691+0000","last_clean":"2023-10-03T17:51:06.375691+0000","last_became_active":"2023-10-03T17:34:42.270194+0000","last_became_peered":"2023-10-03T17:34:42.270194+0000","last_unstale":"2023-10-03T17:51:06.375691+0000","last_undegraded":"2023-10-03T17:51:06.375691+0000","last_fullsized":"2023-10-03T17:51:06.375691+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'8","last_scrub_stamp":"2023-10-03T17:40:24.472573+0000","last_deep_scrub":"46'8","last_deep_scrub_stamp":"2023-10-03T17:40:24.472573+0000","last_clean_scrub_stamp":"2023-10-03T17:40:24.472573+0000","objects_scrubbed":4,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T18:28:04.735033+0000","scrub_duration":0.024990637999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":487,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":43,"num_read_kb":25,"num_write":45,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"46'99","reported_seq":238,"reported_epoch":92,"state":"active+clean","last_fresh":"2023-10-03T17:51:03.439004+0000","last_change":"2023-10-03T17:40:10.452341+0000","last_active":"2023-10-03T17:51:03.439004+0000","last_peered":"2023-10-03T17:51:03.439004+0000","last_clean":"2023-10-03T17:51:03.439004+0000","last_became_active":"2023-10-03T17:34:39.502393+0000","last_became_peered":"2023-10-03T17:34:39.502393+0000","last_unstale":"2023-10-03T17:51:03.439004+0000","last_undegraded":"2023-10-03T17:51:03.439004+0000","last_fullsized":"2023-10-03T17:51:03.439004+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'99","last_scrub_stamp":"2023-10-03T17:40:10.449099+0000","last_deep_scrub":"46'99","last_deep_scrub_stamp":"2023-10-03T17:40:10.449099+0000","last_clean_scrub_stamp":"2023-10-03T17:40:10.449099+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T20:18:04.958666+0000","scrub_duration":0.043225012,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":22072422419,"num_objects":39024,"num_object_clones":33903,"num_object_copies":78048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39024,"num_whiteouts":0,"num_read":374180,"num_read_kb":1416660,"num_write":378458,"num_write_kb":1513826,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3100815360,"data_stored":17157664806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40240,"ondisk_log_size":40240,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":5372,"num_objects":143,"num_object_clones":0,"num_object_copies":286,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":143,"num_whiteouts":0,"num_read":50177,"num_read_kb":41929,"num_write":2063,"num_write_kb":1643,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":983040,"data_stored":10744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7029,"internal_metadata":0},"log_size":11580,"ondisk_log_size":11580,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":7460,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738581,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2407444,"kb_used_data":1514828,"kb_used_omap":2891,"kb_used_meta":889716,"kb_avail":91964396,"statfs":{"total":96636764160,"available":94171541504,"internally_reserved":0,"allocated":1551183872,"data_stored":8581219419,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2960945,"internal_metadata":911069647},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,0,5,1,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 17:51:06 2023","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.623,"15min":0.47799999999999998},"min":{"1min":0.17499999999999999,"5min":0.17199999999999999,"15min":0.14299999999999999},"max":{"1min":2.0310000000000001,"5min":14.787000000000001,"15min":14.787000000000001},"last":0.189},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.61599999999999999,"15min":0.47499999999999998},"min":{"1min":0.20899999999999999,"5min":0.17999999999999999,"15min":0.17100000000000001},"max":{"1min":0.745,"5min":14.919,"15min":14.919},"last":0.24199999999999999}]},{"osd":1,"last update":"Tue Oct 3 17:51:06 2023","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.54600000000000004,"15min":0.52400000000000002},"min":{"1min":0.218,"5min":0.218,"15min":0.16600000000000001},"max":{"1min":4.7069999999999999,"5min":6.7539999999999996,"15min":8.1059999999999999},"last":0.318},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.53200000000000003,"15min":0.46999999999999997},"min":{"1min":0.23699999999999999,"5min":0.223,"15min":0.17699999999999999},"max":{"1min":1.621,"5min":5.194,"15min":5.194},"last":0.21099999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738579,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2093756,"kb_used_data":746676,"kb_used_omap":1,"kb_used_meta":1347070,"kb_avail":92278084,"statfs":{"total":96636764160,"available":94492758016,"internally_reserved":0,"allocated":764596224,"data_stored":5002187774,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":1379400134},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,3,8,11,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 17:51:17 2023","interfaces":[{"interface":"back","average":{"1min":0.317,"5min":0.33200000000000002,"15min":0.33800000000000002},"min":{"1min":0.184,"5min":0.17899999999999999,"15min":0.161},"max":{"1min":0.57799999999999996,"5min":2.7050000000000001,"15min":4.1879999999999997},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.39600000000000002,"5min":0.371,"15min":0.38800000000000001},"min":{"1min":0.19500000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":2.2040000000000002,"5min":5.8959999999999999,"15min":5.8959999999999999},"last":0.24199999999999999}]},{"osd":2,"last update":"Tue Oct 3 17:51:17 2023","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.46300000000000002,"15min":0.40500000000000003},"min":{"1min":0.20200000000000001,"5min":0.17599999999999999,"15min":0.159},"max":{"1min":0.63700000000000001,"5min":4.5519999999999996,"15min":4.5519999999999996},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.34899999999999998,"5min":0.41499999999999998,"15min":0.378},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.16400000000000001},"max":{"1min":0.56399999999999995,"5min":2.6970000000000001,"15min":3.0699999999999998},"last":0.39700000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738579,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2194568,"kb_used_data":768896,"kb_used_omap":1,"kb_used_meta":1425662,"kb_avail":92177272,"statfs":{"total":96636764160,"available":94389526528,"internally_reserved":0,"allocated":787349504,"data_stored":3590143090,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":1459878342},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,4,7,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 17:51:03 2023","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.40799999999999997,"15min":0.40999999999999998},"min":{"1min":0.151,"5min":0.151,"15min":0.151},"max":{"1min":3.9430000000000001,"5min":3.9430000000000001,"15min":5.7649999999999997},"last":0.29799999999999999},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.48599999999999999,"15min":0.47099999999999997},"min":{"1min":0.23000000000000001,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":3.9169999999999998,"5min":3.9169999999999998,"15min":9.4770000000000003},"last":0.28100000000000003}]},{"osd":2,"last update":"Tue Oct 3 17:51:03 2023","interfaces":[{"interface":"back","average":{"1min":0.33100000000000002,"5min":0.46300000000000002,"15min":0.497},"min":{"1min":0.17399999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.95599999999999996,"5min":3.4540000000000002,"15min":9.4149999999999991},"last":0.26100000000000001},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.438,"15min":0.47399999999999998},"min":{"1min":0.217,"5min":0.217,"15min":0.189},"max":{"1min":1.046,"5min":3.4249999999999998,"15min":5.218},"last":0.218}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":3459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":5372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":172032,"data_stored":1913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7029,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":786710528,"data_stored":3579547648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":763473920,"data_stored":5001568275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1550630912,"data_stored":8576548883,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-03T17:51:56.897 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-03T17:34:47.281095+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=34, tm_sec=47, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=40, tm_sec=6, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T17:51:56.898 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T17:52:01.513 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:52:01.509+0000 7f09e4851640 -1 reset not still connected to 0x557d76253d40 2023-10-03T17:52:01.514 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:52:01.513+0000 7f09e4851640 -1 reset not still connected to 0x557d76621040 2023-10-03T17:52:01.514 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:52:01.513+0000 7f09e4851640 -1 reset not still connected to 0x557d7664e750 2023-10-03T17:52:01.514 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:52:01.513+0000 7f09e4851640 -1 reset not still connected to 0x557d7664ea90 2023-10-03T17:52:01.517 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:52:01.517+0000 7f09e4851640 -1 reset not still connected to 0x557d7664f380 2023-10-03T17:52:01.517 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:52:01.517+0000 7f09e4851640 -1 reset not still connected to 0x557d76681ba0 2023-10-03T17:52:01.517 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:52:01.517+0000 7f09e4851640 -1 reset not still connected to 0x557d766b56c0 2023-10-03T17:52:01.517 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:52:01.517+0000 7f09e4851640 -1 reset not still connected to 0x557d76eca270 2023-10-03T17:52:01.517 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:52:01.517+0000 7f09e4851640 -1 reset not still connected to 0x557d76f93ba0 2023-10-03T17:52:07.325 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:52:07.317+0000 7f09e4851640 -1 reset not still connected to 0x557d76621040 2023-10-03T17:52:07.327 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:52:07.325+0000 7f09e4851640 -1 reset not still connected to 0x557d7664e750 2023-10-03T17:52:07.328 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:52:07.325+0000 7f09e4851640 -1 reset not still connected to 0x557d7664ea90 2023-10-03T17:52:07.328 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:52:07.325+0000 7f09e4851640 -1 reset not still connected to 0x557d7664f380 2023-10-03T17:52:07.328 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:52:07.325+0000 7f09e4851640 -1 reset not still connected to 0x557d7664f930 2023-10-03T17:52:07.328 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:52:07.325+0000 7f09e4851640 -1 reset not still connected to 0x557d76681ba0 2023-10-03T17:52:07.329 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:52:07.325+0000 7f09e4851640 -1 reset not still connected to 0x557d766a40d0 2023-10-03T17:52:07.329 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:52:07.325+0000 7f09e4851640 -1 reset not still connected to 0x557d766b56c0 2023-10-03T17:52:07.329 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:52:07.325+0000 7f09e4851640 -1 reset not still connected to 0x557d76eca270 2023-10-03T17:52:07.329 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:52:07.325+0000 7f09e4851640 -1 reset not still connected to 0x557d76f93ba0 2023-10-03T17:52:16.899 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T17:52:16.918 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:52:16.905+0000 7f09e4851640 -1 reset not still connected to 0x557d75f2b2b0 2023-10-03T17:52:16.918 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:52:16.913+0000 7f09e4851640 -1 reset not still connected to 0x557d76253d40 2023-10-03T17:52:16.920 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:52:16.913+0000 7f09e4851640 -1 reset not still connected to 0x557d76621040 2023-10-03T17:52:16.920 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:52:16.913+0000 7f09e4851640 -1 reset not still connected to 0x557d7664e750 2023-10-03T17:52:16.920 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:52:16.913+0000 7f09e4851640 -1 reset not still connected to 0x557d7664ea90 2023-10-03T17:52:16.920 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:52:16.913+0000 7f09e4851640 -1 reset not still connected to 0x557d7664f380 2023-10-03T17:52:16.920 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:52:16.913+0000 7f09e4851640 -1 reset not still connected to 0x557d7664f930 2023-10-03T17:52:16.921 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:52:16.913+0000 7f09e4851640 -1 reset not still connected to 0x557d76681ba0 2023-10-03T17:52:16.921 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:52:16.913+0000 7f09e4851640 -1 reset not still connected to 0x557d766a40d0 2023-10-03T17:52:16.921 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:52:16.913+0000 7f09e4851640 -1 reset not still connected to 0x557d766b56c0 2023-10-03T17:52:16.921 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:52:16.913+0000 7f09e4851640 -1 reset not still connected to 0x557d76eca270 2023-10-03T17:52:16.921 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:52:16.913+0000 7f09e4851640 -1 reset not still connected to 0x557d76f93ba0 2023-10-03T17:52:17.661 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-03T17:52:17.663 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-10-03T17:52:17.741 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":625,"stamp":"2023-10-03T17:52:16.212354+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":22097319727,"num_objects":40539,"num_object_clones":35273,"num_object_copies":81078,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40539,"num_whiteouts":0,"num_read":432885,"num_read_kb":1482183,"num_write":386335,"num_write_kb":1539351,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52162,"ondisk_log_size":52162,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":6867792,"kb_used_data":3074104,"kb_used_omap":2894,"kb_used_meta":3790769,"kb_avail":276247728,"statfs":{"total":289910292480,"available":282877673472,"internally_reserved":0,"allocated":3147882496,"data_stored":18112228679,"data_compressed":7460,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":2964133,"internal_metadata":3881747803},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,1,1,5,9,19,25,11],"upper_bound":512},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":10620928,"num_objects":1289,"num_object_clones":1289,"num_object_copies":2578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1289,"num_whiteouts":0,"num_read":2424,"num_read_kb":9456,"num_write":2438,"num_write_kb":9752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005567"},"pg_stats":[{"pgid":"2.7","version":"97'2119","reported_seq":4590,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-03T17:52:10.806310+0000","last_change":"2023-10-03T17:40:17.407011+0000","last_active":"2023-10-03T17:52:10.806310+0000","last_peered":"2023-10-03T17:52:10.806310+0000","last_clean":"2023-10-03T17:52:10.806310+0000","last_became_active":"2023-10-03T17:34:42.276624+0000","last_became_peered":"2023-10-03T17:34:42.276624+0000","last_unstale":"2023-10-03T17:52:10.806310+0000","last_undegraded":"2023-10-03T17:52:10.806310+0000","last_fullsized":"2023-10-03T17:52:10.806310+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'673","last_scrub_stamp":"2023-10-03T17:40:17.406813+0000","last_deep_scrub":"46'673","last_deep_scrub_stamp":"2023-10-03T17:40:17.406813+0000","last_clean_scrub_stamp":"2023-10-03T17:40:17.406813+0000","objects_scrubbed":7,"log_size":2119,"log_dups_size":0,"ondisk_log_size":2119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T21:38:35.056889+0000","scrub_duration":0.013778076,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":688,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":7475,"num_read_kb":6179,"num_write":331,"num_write_kb":257,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":903,"num_omap_keys":30,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"97'3498","reported_seq":9218,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-03T17:52:10.806469+0000","last_change":"2023-10-03T17:40:13.376519+0000","last_active":"2023-10-03T17:52:10.806469+0000","last_peered":"2023-10-03T17:52:10.806469+0000","last_clean":"2023-10-03T17:52:10.806469+0000","last_became_active":"2023-10-03T17:34:42.271201+0000","last_became_peered":"2023-10-03T17:34:42.271201+0000","last_unstale":"2023-10-03T17:52:10.806469+0000","last_undegraded":"2023-10-03T17:52:10.806469+0000","last_fullsized":"2023-10-03T17:52:10.806469+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'1129","last_scrub_stamp":"2023-10-03T17:40:13.376332+0000","last_deep_scrub":"46'1129","last_deep_scrub_stamp":"2023-10-03T17:40:13.376332+0000","last_clean_scrub_stamp":"2023-10-03T17:40:13.376332+0000","objects_scrubbed":11,"log_size":3498,"log_dups_size":0,"ondisk_log_size":3498,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T18:59:17.088585+0000","scrub_duration":0.015998675,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":623,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":14815,"num_read_kb":12646,"num_write":544,"num_write_kb":440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2912,"num_omap_keys":93,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"96'59","reported_seq":204,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-03T17:52:03.303639+0000","last_change":"2023-10-03T17:40:25.496010+0000","last_active":"2023-10-03T17:52:03.303639+0000","last_peered":"2023-10-03T17:52:03.303639+0000","last_clean":"2023-10-03T17:52:03.303639+0000","last_became_active":"2023-10-03T17:34:42.276941+0000","last_became_peered":"2023-10-03T17:34:42.276941+0000","last_unstale":"2023-10-03T17:52:03.303639+0000","last_undegraded":"2023-10-03T17:52:03.303639+0000","last_fullsized":"2023-10-03T17:52:03.303639+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'14","last_scrub_stamp":"2023-10-03T17:40:25.495827+0000","last_deep_scrub":"46'14","last_deep_scrub_stamp":"2023-10-03T17:40:25.495827+0000","last_clean_scrub_stamp":"2023-10-03T17:40:25.495827+0000","objects_scrubbed":7,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:05:44.939663+0000","scrub_duration":0.0133078,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1107,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":53,"num_read_kb":26,"num_write":52,"num_write_kb":38,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"97'2800","reported_seq":6918,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-03T17:52:10.806349+0000","last_change":"2023-10-03T17:40:26.513902+0000","last_active":"2023-10-03T17:52:10.806349+0000","last_peered":"2023-10-03T17:52:10.806349+0000","last_clean":"2023-10-03T17:52:10.806349+0000","last_became_active":"2023-10-03T17:34:42.276459+0000","last_became_peered":"2023-10-03T17:34:42.276459+0000","last_unstale":"2023-10-03T17:52:10.806349+0000","last_undegraded":"2023-10-03T17:52:10.806349+0000","last_fullsized":"2023-10-03T17:52:10.806349+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'964","last_scrub_stamp":"2023-10-03T17:40:26.513729+0000","last_deep_scrub":"46'964","last_deep_scrub_stamp":"2023-10-03T17:40:26.513729+0000","last_clean_scrub_stamp":"2023-10-03T17:40:26.513729+0000","objects_scrubbed":10,"log_size":2800,"log_dups_size":0,"ondisk_log_size":2800,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:41:33.182751+0000","scrub_duration":0.018190280999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":335,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":10886,"num_read_kb":9110,"num_write":412,"num_write_kb":333,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1731,"num_omap_keys":80,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"97'128789","reported_seq":209715,"reported_epoch":97,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T17:52:13.798364+0000","last_change":"2023-10-03T17:52:10.827734+0000","last_active":"2023-10-03T17:52:13.798364+0000","last_peered":"2023-10-03T17:52:13.798364+0000","last_clean":"2023-10-03T17:52:13.798364+0000","last_became_active":"2023-10-03T17:34:48.310701+0000","last_became_peered":"2023-10-03T17:34:48.310701+0000","last_unstale":"2023-10-03T17:52:13.798364+0000","last_undegraded":"2023-10-03T17:52:13.798364+0000","last_fullsized":"2023-10-03T17:52:13.798364+0000","mapping_epoch":16,"log_start":"88'118698","ondisk_log_start":"88'118698","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'80695","last_scrub_stamp":"2023-10-03T17:46:13.982571+0000","last_deep_scrub":"66'80695","last_deep_scrub_stamp":"2023-10-03T17:46:13.982571+0000","last_clean_scrub_stamp":"2023-10-03T17:46:13.982571+0000","objects_scrubbed":3410,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":62,"last_scrub_duration":363,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:15:57.360096+0000","scrub_duration":362.26574615599998,"objects_trimmed":157,"snaptrim_duration":7.3672095500000001,"stat_sum":{"num_bytes":5695791104,"num_objects":9739,"num_object_clones":8419,"num_object_copies":19478,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9739,"num_whiteouts":0,"num_read":97975,"num_read_kb":370984,"num_write":99310,"num_write_kb":397240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"54","length":"1e"},{"start":"7c","length":"8"},{"start":"8a","length":"2"},{"start":"90","length":"a"},{"start":"a4","length":"a"},{"start":"b8","length":"a"},{"start":"cc","length":"a"},{"start":"e0","length":"a"},{"start":"f4","length":"8"}]},{"pgid":"2.2","version":"97'2893","reported_seq":6178,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-03T17:52:10.806333+0000","last_change":"2023-10-03T17:40:08.675119+0000","last_active":"2023-10-03T17:52:10.806333+0000","last_peered":"2023-10-03T17:52:10.806333+0000","last_clean":"2023-10-03T17:52:10.806333+0000","last_became_active":"2023-10-03T17:34:42.270625+0000","last_became_peered":"2023-10-03T17:34:42.270625+0000","last_unstale":"2023-10-03T17:52:10.806333+0000","last_undegraded":"2023-10-03T17:52:10.806333+0000","last_fullsized":"2023-10-03T17:52:10.806333+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'841","last_scrub_stamp":"2023-10-03T17:40:08.674910+0000","last_deep_scrub":"46'841","last_deep_scrub_stamp":"2023-10-03T17:40:08.674910+0000","last_clean_scrub_stamp":"2023-10-03T17:40:08.674910+0000","objects_scrubbed":11,"log_size":2893,"log_dups_size":0,"ondisk_log_size":2893,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T21:05:42.790614+0000","scrub_duration":0.039189555000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":808,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":9988,"num_read_kb":8272,"num_write":442,"num_write_kb":352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2564,"num_omap_keys":60,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"97'113535","reported_seq":192347,"reported_epoch":97,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T17:52:13.873038+0000","last_change":"2023-10-03T17:43:28.494717+0000","last_active":"2023-10-03T17:52:13.873038+0000","last_peered":"2023-10-03T17:52:13.873038+0000","last_clean":"2023-10-03T17:52:13.873038+0000","last_became_active":"2023-10-03T17:34:48.310555+0000","last_became_peered":"2023-10-03T17:34:48.310555+0000","last_unstale":"2023-10-03T17:52:13.873038+0000","last_undegraded":"2023-10-03T17:52:13.873038+0000","last_fullsized":"2023-10-03T17:52:13.873038+0000","mapping_epoch":16,"log_start":"88'103532","ondisk_log_start":"88'103532","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_clean_scrub_stamp":"2023-10-03T17:34:47.281095+0000","objects_scrubbed":8229,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":160,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 525s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086289699999999995,"stat_sum":{"num_bytes":5357834240,"num_objects":15167,"num_object_clones":13925,"num_object_copies":30334,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15167,"num_whiteouts":0,"num_read":92066,"num_read_kb":347840,"num_write":92464,"num_write_kb":369856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"97'272","reported_seq":2724,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-03T17:52:11.210961+0000","last_change":"2023-10-03T17:40:17.641493+0000","last_active":"2023-10-03T17:52:11.210961+0000","last_peered":"2023-10-03T17:52:11.210961+0000","last_clean":"2023-10-03T17:52:11.210961+0000","last_became_active":"2023-10-03T17:34:42.278196+0000","last_became_peered":"2023-10-03T17:34:42.278196+0000","last_unstale":"2023-10-03T17:52:11.210961+0000","last_undegraded":"2023-10-03T17:52:11.210961+0000","last_fullsized":"2023-10-03T17:52:11.210961+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'30","last_scrub_stamp":"2023-10-03T17:40:17.640969+0000","last_deep_scrub":"46'30","last_deep_scrub_stamp":"2023-10-03T17:40:17.640969+0000","last_clean_scrub_stamp":"2023-10-03T17:40:17.640969+0000","objects_scrubbed":6,"log_size":272,"log_dups_size":0,"ondisk_log_size":272,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:04:02.954896+0000","scrub_duration":0.071054565,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":522,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":7444,"num_read_kb":6157,"num_write":305,"num_write_kb":242,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":903,"num_omap_keys":30,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"97'127536","reported_seq":204374,"reported_epoch":97,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T17:52:13.767078+0000","last_change":"2023-10-03T17:52:10.209336+0000","last_active":"2023-10-03T17:52:13.767078+0000","last_peered":"2023-10-03T17:52:13.767078+0000","last_clean":"2023-10-03T17:52:13.767078+0000","last_became_active":"2023-10-03T17:34:48.310912+0000","last_became_peered":"2023-10-03T17:34:48.310912+0000","last_unstale":"2023-10-03T17:52:13.767078+0000","last_undegraded":"2023-10-03T17:52:13.767078+0000","last_fullsized":"2023-10-03T17:52:13.767078+0000","mapping_epoch":16,"log_start":"88'117455","ondisk_log_start":"88'117455","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'58196","last_scrub_stamp":"2023-10-03T17:43:26.193075+0000","last_deep_scrub":"53'58196","last_deep_scrub_stamp":"2023-10-03T17:43:26.193075+0000","last_clean_scrub_stamp":"2023-10-03T17:43:26.193075+0000","objects_scrubbed":2326,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":25,"last_scrub_duration":196,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T19:30:13.881336+0000","scrub_duration":195.44739730800001,"objects_trimmed":152,"snaptrim_duration":6.7492099840000002,"stat_sum":{"num_bytes":5488234496,"num_objects":7092,"num_object_clones":5821,"num_object_copies":14184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7092,"num_whiteouts":0,"num_read":94292,"num_read_kb":357896,"num_write":95474,"num_write_kb":381896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"54","length":"1e"},{"start":"7c","length":"8"},{"start":"8a","length":"2"},{"start":"90","length":"a"},{"start":"a4","length":"a"},{"start":"b8","length":"a"},{"start":"cc","length":"a"},{"start":"e0","length":"a"},{"start":"f4","length":"a"},{"start":"108","length":"a"},{"start":"11c","length":"a"},{"start":"130","length":"a"},{"start":"143","length":"5"}]},{"pgid":"2.0","version":"96'144","reported_seq":1070,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-03T17:52:05.707074+0000","last_change":"2023-10-03T17:40:10.708344+0000","last_active":"2023-10-03T17:52:05.707074+0000","last_peered":"2023-10-03T17:52:05.707074+0000","last_clean":"2023-10-03T17:52:05.707074+0000","last_became_active":"2023-10-03T17:34:42.272306+0000","last_became_peered":"2023-10-03T17:34:42.272306+0000","last_unstale":"2023-10-03T17:52:05.707074+0000","last_undegraded":"2023-10-03T17:52:05.707074+0000","last_fullsized":"2023-10-03T17:52:05.707074+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'20","last_scrub_stamp":"2023-10-03T17:40:10.708060+0000","last_deep_scrub":"46'20","last_deep_scrub_stamp":"2023-10-03T17:40:10.708060+0000","last_clean_scrub_stamp":"2023-10-03T17:40:10.708060+0000","objects_scrubbed":7,"log_size":144,"log_dups_size":0,"ondisk_log_size":144,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:03:09.097252+0000","scrub_duration":0.042279491000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":904,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":2536,"num_read_kb":2083,"num_write":150,"num_write_kb":113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":301,"num_omap_keys":10,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"97'127983","reported_seq":206066,"reported_epoch":97,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T17:52:13.893506+0000","last_change":"2023-10-03T17:52:13.147369+0000","last_active":"2023-10-03T17:52:13.893506+0000","last_peered":"2023-10-03T17:52:13.893506+0000","last_clean":"2023-10-03T17:52:13.893506+0000","last_became_active":"2023-10-03T17:34:48.310612+0000","last_became_peered":"2023-10-03T17:34:48.310612+0000","last_unstale":"2023-10-03T17:52:13.893506+0000","last_undegraded":"2023-10-03T17:52:13.893506+0000","last_fullsized":"2023-10-03T17:52:13.893506+0000","mapping_epoch":16,"log_start":"88'117931","ondisk_log_start":"88'117931","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'75807","last_scrub_stamp":"2023-10-03T17:45:43.833310+0000","last_deep_scrub":"61'75807","last_deep_scrub_stamp":"2023-10-03T17:45:43.833310+0000","last_clean_scrub_stamp":"2023-10-03T17:45:43.833310+0000","objects_scrubbed":3607,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":43,"last_scrub_duration":315,"scrub_schedule":"queued for deep scrub","scrub_duration":314.26374841099999,"objects_trimmed":147,"snaptrim_duration":9.6731426789999997,"stat_sum":{"num_bytes":5554864147,"num_objects":8396,"num_object_clones":7108,"num_object_copies":16792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8396,"num_whiteouts":0,"num_read":95204,"num_read_kb":360872,"num_write":96667,"num_write_kb":386662,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"54","length":"1e"},{"start":"7c","length":"8"},{"start":"8a","length":"2"},{"start":"90","length":"a"},{"start":"a4","length":"a"},{"start":"b8","length":"a"},{"start":"cc","length":"a"},{"start":"e0","length":"a"},{"start":"f4","length":"a"},{"start":"108","length":"a"},{"start":"11c","length":"7"}]},{"pgid":"2.3","version":"96'51","reported_seq":194,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-03T17:52:02.561058+0000","last_change":"2023-10-03T17:40:24.472734+0000","last_active":"2023-10-03T17:52:02.561058+0000","last_peered":"2023-10-03T17:52:02.561058+0000","last_clean":"2023-10-03T17:52:02.561058+0000","last_became_active":"2023-10-03T17:34:42.270194+0000","last_became_peered":"2023-10-03T17:34:42.270194+0000","last_unstale":"2023-10-03T17:52:02.561058+0000","last_undegraded":"2023-10-03T17:52:02.561058+0000","last_fullsized":"2023-10-03T17:52:02.561058+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'8","last_scrub_stamp":"2023-10-03T17:40:24.472573+0000","last_deep_scrub":"46'8","last_deep_scrub_stamp":"2023-10-03T17:40:24.472573+0000","last_clean_scrub_stamp":"2023-10-03T17:40:24.472573+0000","objects_scrubbed":4,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T18:28:04.735033+0000","scrub_duration":0.024990637999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":385,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":45,"num_read_kb":27,"num_write":47,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"46'99","reported_seq":242,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-03T17:52:02.449635+0000","last_change":"2023-10-03T17:40:10.452341+0000","last_active":"2023-10-03T17:52:02.449635+0000","last_peered":"2023-10-03T17:52:02.449635+0000","last_clean":"2023-10-03T17:52:02.449635+0000","last_became_active":"2023-10-03T17:34:39.502393+0000","last_became_peered":"2023-10-03T17:34:39.502393+0000","last_unstale":"2023-10-03T17:52:02.449635+0000","last_undegraded":"2023-10-03T17:52:02.449635+0000","last_fullsized":"2023-10-03T17:52:02.449635+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'99","last_scrub_stamp":"2023-10-03T17:40:10.449099+0000","last_deep_scrub":"46'99","last_deep_scrub_stamp":"2023-10-03T17:40:10.449099+0000","last_clean_scrub_stamp":"2023-10-03T17:40:10.449099+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T20:18:04.958666+0000","scrub_duration":0.043225012,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":22096723987,"num_objects":40394,"num_object_clones":35273,"num_object_copies":80788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40394,"num_whiteouts":0,"num_read":379537,"num_read_kb":1437592,"num_write":383915,"num_write_kb":1535654,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3145637888,"data_stored":18144067622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40227,"ondisk_log_size":40227,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":5372,"num_objects":143,"num_object_clones":0,"num_object_copies":286,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":143,"num_whiteouts":0,"num_read":53242,"num_read_kb":44500,"num_write":2283,"num_write_kb":1803,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":983040,"data_stored":10744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7029,"internal_metadata":0},"log_size":11836,"ondisk_log_size":11836,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":7460,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738585,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2491672,"kb_used_data":1536592,"kb_used_omap":2891,"kb_used_meta":952180,"kb_avail":91880168,"statfs":{"total":96636764160,"available":94085292032,"internally_reserved":0,"allocated":1573470208,"data_stored":9062827721,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2960945,"internal_metadata":975032783},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,4,8,12,6],"upper_bound":512},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 17:52:06 2023","interfaces":[{"interface":"back","average":{"1min":0.32000000000000001,"5min":0.60399999999999998,"15min":0.47199999999999998},"min":{"1min":0.158,"5min":0.158,"15min":0.158},"max":{"1min":0.57999999999999996,"5min":14.787000000000001,"15min":14.787000000000001},"last":0.22900000000000001},{"interface":"front","average":{"1min":0.30499999999999999,"5min":0.59599999999999997,"15min":0.46700000000000003},"min":{"1min":0.186,"5min":0.185,"15min":0.17100000000000001},"max":{"1min":0.47199999999999998,"5min":14.919,"15min":14.919},"last":0.28899999999999998}]},{"osd":1,"last update":"Tue Oct 3 17:52:06 2023","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.58799999999999997,"15min":0.53200000000000003},"min":{"1min":0.23300000000000001,"5min":0.218,"15min":0.17599999999999999},"max":{"1min":3.5659999999999998,"5min":6.7539999999999996,"15min":8.1059999999999999},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.34899999999999998,"5min":0.51100000000000001,"15min":0.46500000000000002},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.17699999999999999},"max":{"1min":0.62,"5min":5.194,"15min":5.194},"last":0.252}]}]},{"osd":1,"up_from":8,"seq":34359738583,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2139428,"kb_used_data":757660,"kb_used_omap":1,"kb_used_meta":1381758,"kb_avail":92232412,"statfs":{"total":96636764160,"available":94445989888,"internally_reserved":0,"allocated":775843840,"data_stored":5278422738,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":1414920646},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,1,1,0,3,3,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 17:51:17 2023","interfaces":[{"interface":"back","average":{"1min":0.317,"5min":0.33200000000000002,"15min":0.33800000000000002},"min":{"1min":0.184,"5min":0.17899999999999999,"15min":0.161},"max":{"1min":0.57799999999999996,"5min":2.7050000000000001,"15min":4.1879999999999997},"last":0.245},{"interface":"front","average":{"1min":0.39600000000000002,"5min":0.371,"15min":0.38800000000000001},"min":{"1min":0.19500000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":2.2040000000000002,"5min":5.8959999999999999,"15min":5.8959999999999999},"last":1.913}]},{"osd":2,"last update":"Tue Oct 3 17:51:17 2023","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.46300000000000002,"15min":0.40500000000000003},"min":{"1min":0.20200000000000001,"5min":0.17599999999999999,"15min":0.159},"max":{"1min":0.63700000000000001,"5min":4.5519999999999996,"15min":4.5519999999999996},"last":0.20999999999999999},{"interface":"front","average":{"1min":0.34899999999999998,"5min":0.41499999999999998,"15min":0.378},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.16400000000000001},"max":{"1min":0.56399999999999995,"5min":2.6970000000000001,"15min":3.0699999999999998},"last":0.27400000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738583,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2236692,"kb_used_data":779852,"kb_used_omap":1,"kb_used_meta":1456830,"kb_avail":92135148,"statfs":{"total":96636764160,"available":94346391552,"internally_reserved":0,"allocated":798568448,"data_stored":3770978220,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":1491794374},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,2,8,10,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 17:52:04 2023","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.39600000000000002,"15min":0.41499999999999998},"min":{"1min":0.182,"5min":0.151,"15min":0.151},"max":{"1min":2.8340000000000001,"5min":3.9430000000000001,"15min":5.7649999999999997},"last":0.26700000000000002},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.49299999999999999,"15min":0.47999999999999998},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.16900000000000001},"max":{"1min":3.6930000000000001,"5min":3.9169999999999998,"15min":9.4770000000000003},"last":0.28499999999999998}]},{"osd":2,"last update":"Tue Oct 3 17:52:04 2023","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.47199999999999998,"15min":0.51100000000000001},"min":{"1min":0.215,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":3.996,"5min":3.996,"15min":9.4149999999999991},"last":0.218},{"interface":"front","average":{"1min":0.31900000000000001,"5min":0.36799999999999999,"15min":0.46899999999999997},"min":{"1min":0.19600000000000001,"5min":0.19600000000000001,"15min":0.189},"max":{"1min":0.64900000000000002,"5min":1.046,"15min":5.218},"last":0.249}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":3561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":5372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7029,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":797839360,"data_stored":3779305472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":774688768,"data_stored":5283532819,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1573109760,"data_stored":9081229331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-03T17:52:17.743 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-03T17:34:47.281095+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=34, tm_sec=47, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=40, tm_sec=6, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T17:52:17.744 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T17:52:29.661 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:52:29.661+0000 7f09e4851640 -1 reset not still connected to 0x557d75f2b2b0 2023-10-03T17:52:29.661 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:52:29.661+0000 7f09e4851640 -1 reset not still connected to 0x557d76253d40 2023-10-03T17:52:29.662 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:52:29.661+0000 7f09e4851640 -1 reset not still connected to 0x557d76621040 2023-10-03T17:52:29.662 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:52:29.661+0000 7f09e4851640 -1 reset not still connected to 0x557d7664e750 2023-10-03T17:52:29.662 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:52:29.661+0000 7f09e4851640 -1 reset not still connected to 0x557d7664ea90 2023-10-03T17:52:30.009 INFO:tasks.ceph.mgr.x.smithi111.stderr:2023-10-03T17:52:30.013+0000 7fe48bb97640 -1 calc_pg_upmaps abort due to max <= 0 2023-10-03T17:52:37.744 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T17:52:38.567 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-03T17:52:38.568 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-10-03T17:52:38.593 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":636,"stamp":"2023-10-03T17:52:38.221448+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":22121498415,"num_objects":41554,"num_object_clones":36288,"num_object_copies":83108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41554,"num_whiteouts":0,"num_read":439575,"num_read_kb":1504006,"num_write":391693,"num_write_kb":1560783,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52242,"ondisk_log_size":52242,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":7112960,"kb_used_data":3117160,"kb_used_omap":2894,"kb_used_meta":3992881,"kb_avail":276002560,"statfs":{"total":289910292480,"available":282626621440,"internally_reserved":0,"allocated":3191971840,"data_stored":18843422023,"data_compressed":7460,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":2964133,"internal_metadata":4088710491},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,3,4,7,19,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":12849152,"num_objects":315,"num_object_clones":315,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":315,"num_whiteouts":0,"num_read":2750,"num_read_kb":10796,"num_write":2812,"num_write_kb":11248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007361"},"pg_stats":[{"pgid":"2.7","version":"97'2136","reported_seq":4652,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-03T17:52:31.286214+0000","last_change":"2023-10-03T17:40:17.407011+0000","last_active":"2023-10-03T17:52:31.286214+0000","last_peered":"2023-10-03T17:52:31.286214+0000","last_clean":"2023-10-03T17:52:31.286214+0000","last_became_active":"2023-10-03T17:34:42.276624+0000","last_became_peered":"2023-10-03T17:34:42.276624+0000","last_unstale":"2023-10-03T17:52:31.286214+0000","last_undegraded":"2023-10-03T17:52:31.286214+0000","last_fullsized":"2023-10-03T17:52:31.286214+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'673","last_scrub_stamp":"2023-10-03T17:40:17.406813+0000","last_deep_scrub":"46'673","last_deep_scrub_stamp":"2023-10-03T17:40:17.406813+0000","last_clean_scrub_stamp":"2023-10-03T17:40:17.406813+0000","objects_scrubbed":7,"log_size":2136,"log_dups_size":0,"ondisk_log_size":2136,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T21:38:35.056889+0000","scrub_duration":0.013778076,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":688,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":7673,"num_read_kb":6341,"num_write":331,"num_write_kb":257,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":903,"num_omap_keys":30,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"97'3524","reported_seq":9361,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-03T17:52:31.286446+0000","last_change":"2023-10-03T17:40:13.376519+0000","last_active":"2023-10-03T17:52:31.286446+0000","last_peered":"2023-10-03T17:52:31.286446+0000","last_clean":"2023-10-03T17:52:31.286446+0000","last_became_active":"2023-10-03T17:34:42.271201+0000","last_became_peered":"2023-10-03T17:34:42.271201+0000","last_unstale":"2023-10-03T17:52:31.286446+0000","last_undegraded":"2023-10-03T17:52:31.286446+0000","last_fullsized":"2023-10-03T17:52:31.286446+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'1129","last_scrub_stamp":"2023-10-03T17:40:13.376332+0000","last_deep_scrub":"46'1129","last_deep_scrub_stamp":"2023-10-03T17:40:13.376332+0000","last_clean_scrub_stamp":"2023-10-03T17:40:13.376332+0000","objects_scrubbed":11,"log_size":3524,"log_dups_size":0,"ondisk_log_size":3524,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T18:59:17.088585+0000","scrub_duration":0.015998675,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":623,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":15229,"num_read_kb":12999,"num_write":544,"num_write_kb":440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2912,"num_omap_keys":93,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"96'59","reported_seq":204,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-03T17:52:03.303639+0000","last_change":"2023-10-03T17:40:25.496010+0000","last_active":"2023-10-03T17:52:03.303639+0000","last_peered":"2023-10-03T17:52:03.303639+0000","last_clean":"2023-10-03T17:52:03.303639+0000","last_became_active":"2023-10-03T17:34:42.276941+0000","last_became_peered":"2023-10-03T17:34:42.276941+0000","last_unstale":"2023-10-03T17:52:03.303639+0000","last_undegraded":"2023-10-03T17:52:03.303639+0000","last_fullsized":"2023-10-03T17:52:03.303639+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'14","last_scrub_stamp":"2023-10-03T17:40:25.495827+0000","last_deep_scrub":"46'14","last_deep_scrub_stamp":"2023-10-03T17:40:25.495827+0000","last_clean_scrub_stamp":"2023-10-03T17:40:25.495827+0000","objects_scrubbed":7,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:05:44.939663+0000","scrub_duration":0.0133078,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1107,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":53,"num_read_kb":26,"num_write":52,"num_write_kb":38,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"97'2823","reported_seq":7023,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-03T17:52:31.286224+0000","last_change":"2023-10-03T17:40:26.513902+0000","last_active":"2023-10-03T17:52:31.286224+0000","last_peered":"2023-10-03T17:52:31.286224+0000","last_clean":"2023-10-03T17:52:31.286224+0000","last_became_active":"2023-10-03T17:34:42.276459+0000","last_became_peered":"2023-10-03T17:34:42.276459+0000","last_unstale":"2023-10-03T17:52:31.286224+0000","last_undegraded":"2023-10-03T17:52:31.286224+0000","last_fullsized":"2023-10-03T17:52:31.286224+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'964","last_scrub_stamp":"2023-10-03T17:40:26.513729+0000","last_deep_scrub":"46'964","last_deep_scrub_stamp":"2023-10-03T17:40:26.513729+0000","last_clean_scrub_stamp":"2023-10-03T17:40:26.513729+0000","objects_scrubbed":10,"log_size":2823,"log_dups_size":0,"ondisk_log_size":2823,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:41:33.182751+0000","scrub_duration":0.018190280999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":335,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":11172,"num_read_kb":9347,"num_write":412,"num_write_kb":333,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1731,"num_omap_keys":80,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"97'131186","reported_seq":212902,"reported_epoch":97,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T17:52:34.023082+0000","last_change":"2023-10-03T17:52:31.689938+0000","last_active":"2023-10-03T17:52:34.023082+0000","last_peered":"2023-10-03T17:52:34.023082+0000","last_clean":"2023-10-03T17:52:34.023082+0000","last_became_active":"2023-10-03T17:34:48.310701+0000","last_became_peered":"2023-10-03T17:34:48.310701+0000","last_unstale":"2023-10-03T17:52:34.023082+0000","last_undegraded":"2023-10-03T17:52:34.023082+0000","last_fullsized":"2023-10-03T17:52:34.023082+0000","mapping_epoch":16,"log_start":"93'121106","ondisk_log_start":"93'121106","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'80695","last_scrub_stamp":"2023-10-03T17:46:13.982571+0000","last_deep_scrub":"66'80695","last_deep_scrub_stamp":"2023-10-03T17:46:13.982571+0000","last_clean_scrub_stamp":"2023-10-03T17:46:13.982571+0000","objects_scrubbed":3410,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":58,"last_scrub_duration":363,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:15:57.360096+0000","scrub_duration":362.26574615599998,"objects_trimmed":409,"snaptrim_duration":28.229019962999999,"stat_sum":{"num_bytes":5701873664,"num_objects":9975,"num_object_clones":8655,"num_object_copies":19950,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9975,"num_whiteouts":0,"num_read":99350,"num_read_kb":376372,"num_write":100715,"num_write_kb":402860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"54","length":"1e"},{"start":"7c","length":"8"},{"start":"8a","length":"2"},{"start":"90","length":"a"},{"start":"a4","length":"a"},{"start":"b8","length":"a"},{"start":"cc","length":"a"},{"start":"e0","length":"a"},{"start":"f4","length":"a"},{"start":"108","length":"2"}]},{"pgid":"2.2","version":"97'2921","reported_seq":6268,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-03T17:52:33.573777+0000","last_change":"2023-10-03T17:40:08.675119+0000","last_active":"2023-10-03T17:52:33.573777+0000","last_peered":"2023-10-03T17:52:33.573777+0000","last_clean":"2023-10-03T17:52:33.573777+0000","last_became_active":"2023-10-03T17:34:42.270625+0000","last_became_peered":"2023-10-03T17:34:42.270625+0000","last_unstale":"2023-10-03T17:52:33.573777+0000","last_undegraded":"2023-10-03T17:52:33.573777+0000","last_fullsized":"2023-10-03T17:52:33.573777+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'841","last_scrub_stamp":"2023-10-03T17:40:08.674910+0000","last_deep_scrub":"46'841","last_deep_scrub_stamp":"2023-10-03T17:40:08.674910+0000","last_clean_scrub_stamp":"2023-10-03T17:40:08.674910+0000","objects_scrubbed":11,"log_size":2921,"log_dups_size":0,"ondisk_log_size":2921,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T21:05:42.790614+0000","scrub_duration":0.039189555000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":808,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":10254,"num_read_kb":8491,"num_write":442,"num_write_kb":352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2564,"num_omap_keys":60,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"97'115538","reported_seq":195228,"reported_epoch":97,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T17:52:34.005987+0000","last_change":"2023-10-03T17:43:28.494717+0000","last_active":"2023-10-03T17:52:34.005987+0000","last_peered":"2023-10-03T17:52:34.005987+0000","last_clean":"2023-10-03T17:52:34.005987+0000","last_became_active":"2023-10-03T17:34:48.310555+0000","last_became_peered":"2023-10-03T17:34:48.310555+0000","last_unstale":"2023-10-03T17:52:34.005987+0000","last_undegraded":"2023-10-03T17:52:34.005987+0000","last_fullsized":"2023-10-03T17:52:34.005987+0000","mapping_epoch":16,"log_start":"88'105534","ondisk_log_start":"88'105534","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_clean_scrub_stamp":"2023-10-03T17:34:47.281095+0000","objects_scrubbed":8465,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":160,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 545s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086289699999999995,"stat_sum":{"num_bytes":5364518912,"num_objects":15621,"num_object_clones":14379,"num_object_copies":31242,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15621,"num_whiteouts":0,"num_read":93387,"num_read_kb":353008,"num_write":93797,"num_write_kb":375188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"97'272","reported_seq":2769,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-03T17:52:31.400886+0000","last_change":"2023-10-03T17:40:17.641493+0000","last_active":"2023-10-03T17:52:31.400886+0000","last_peered":"2023-10-03T17:52:31.400886+0000","last_clean":"2023-10-03T17:52:31.400886+0000","last_became_active":"2023-10-03T17:34:42.278196+0000","last_became_peered":"2023-10-03T17:34:42.278196+0000","last_unstale":"2023-10-03T17:52:31.400886+0000","last_undegraded":"2023-10-03T17:52:31.400886+0000","last_fullsized":"2023-10-03T17:52:31.400886+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'30","last_scrub_stamp":"2023-10-03T17:40:17.640969+0000","last_deep_scrub":"46'30","last_deep_scrub_stamp":"2023-10-03T17:40:17.640969+0000","last_clean_scrub_stamp":"2023-10-03T17:40:17.640969+0000","objects_scrubbed":6,"log_size":272,"log_dups_size":0,"ondisk_log_size":272,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:04:02.954896+0000","scrub_duration":0.071054565,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":522,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":7642,"num_read_kb":6319,"num_write":305,"num_write_kb":242,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":903,"num_omap_keys":30,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"97'129939","reported_seq":207471,"reported_epoch":97,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T17:52:34.015620+0000","last_change":"2023-10-03T17:52:32.770575+0000","last_active":"2023-10-03T17:52:34.015620+0000","last_peered":"2023-10-03T17:52:34.015620+0000","last_clean":"2023-10-03T17:52:34.015620+0000","last_became_active":"2023-10-03T17:34:48.310912+0000","last_became_peered":"2023-10-03T17:34:48.310912+0000","last_unstale":"2023-10-03T17:52:34.015620+0000","last_undegraded":"2023-10-03T17:52:34.015620+0000","last_fullsized":"2023-10-03T17:52:34.015620+0000","mapping_epoch":16,"log_start":"93'119865","ondisk_log_start":"93'119865","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'58196","last_scrub_stamp":"2023-10-03T17:43:26.193075+0000","last_deep_scrub":"53'58196","last_deep_scrub_stamp":"2023-10-03T17:43:26.193075+0000","last_clean_scrub_stamp":"2023-10-03T17:43:26.193075+0000","objects_scrubbed":2326,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":20,"last_scrub_duration":196,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T19:30:13.881336+0000","scrub_duration":195.44739730800001,"objects_trimmed":434,"snaptrim_duration":29.310435538,"stat_sum":{"num_bytes":5494448128,"num_objects":7283,"num_object_clones":6012,"num_object_copies":14566,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7283,"num_whiteouts":0,"num_read":95577,"num_read_kb":362936,"num_write":96840,"num_write_kb":387360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"54","length":"1e"},{"start":"7c","length":"8"},{"start":"8a","length":"2"},{"start":"90","length":"a"},{"start":"a4","length":"a"},{"start":"b8","length":"a"},{"start":"cc","length":"a"},{"start":"e0","length":"a"},{"start":"f4","length":"a"},{"start":"108","length":"a"},{"start":"11c","length":"a"},{"start":"130","length":"a"},{"start":"143","length":"7"},{"start":"150","length":"1"},{"start":"152","length":"1"},{"start":"154","length":"1"}]},{"pgid":"2.0","version":"96'144","reported_seq":1085,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-03T17:52:26.167470+0000","last_change":"2023-10-03T17:40:10.708344+0000","last_active":"2023-10-03T17:52:26.167470+0000","last_peered":"2023-10-03T17:52:26.167470+0000","last_clean":"2023-10-03T17:52:26.167470+0000","last_became_active":"2023-10-03T17:34:42.272306+0000","last_became_peered":"2023-10-03T17:34:42.272306+0000","last_unstale":"2023-10-03T17:52:26.167470+0000","last_undegraded":"2023-10-03T17:52:26.167470+0000","last_fullsized":"2023-10-03T17:52:26.167470+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'20","last_scrub_stamp":"2023-10-03T17:40:10.708060+0000","last_deep_scrub":"46'20","last_deep_scrub_stamp":"2023-10-03T17:40:10.708060+0000","last_clean_scrub_stamp":"2023-10-03T17:40:10.708060+0000","objects_scrubbed":7,"log_size":144,"log_dups_size":0,"ondisk_log_size":144,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:03:09.097252+0000","scrub_duration":0.042279491000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":904,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":2602,"num_read_kb":2137,"num_write":150,"num_write_kb":113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":301,"num_omap_keys":10,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"97'130301","reported_seq":209095,"reported_epoch":97,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T17:52:34.017847+0000","last_change":"2023-10-03T17:52:33.589214+0000","last_active":"2023-10-03T17:52:34.017847+0000","last_peered":"2023-10-03T17:52:34.017847+0000","last_clean":"2023-10-03T17:52:34.017847+0000","last_became_active":"2023-10-03T17:34:48.310612+0000","last_became_peered":"2023-10-03T17:34:48.310612+0000","last_unstale":"2023-10-03T17:52:34.017847+0000","last_undegraded":"2023-10-03T17:52:34.017847+0000","last_fullsized":"2023-10-03T17:52:34.017847+0000","mapping_epoch":16,"log_start":"93'120246","ondisk_log_start":"93'120246","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'75807","last_scrub_stamp":"2023-10-03T17:45:43.833310+0000","last_deep_scrub":"61'75807","last_deep_scrub_stamp":"2023-10-03T17:45:43.833310+0000","last_clean_scrub_stamp":"2023-10-03T17:45:43.833310+0000","objects_scrubbed":3607,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":38,"last_scrub_duration":315,"scrub_schedule":"queued for deep scrub","scrub_duration":314.26374841099999,"objects_trimmed":457,"snaptrim_duration":30.121752544,"stat_sum":{"num_bytes":5560061971,"num_objects":8530,"num_object_clones":7242,"num_object_copies":17060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8530,"num_whiteouts":0,"num_read":96485,"num_read_kb":365912,"num_write":97921,"num_write_kb":391678,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"54","length":"1e"},{"start":"7c","length":"8"},{"start":"8a","length":"2"},{"start":"90","length":"a"},{"start":"a4","length":"a"},{"start":"b8","length":"a"},{"start":"cc","length":"a"},{"start":"e0","length":"a"},{"start":"f4","length":"a"},{"start":"108","length":"a"},{"start":"11c","length":"a"},{"start":"130","length":"2"}]},{"pgid":"2.3","version":"96'51","reported_seq":194,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-03T17:52:02.561058+0000","last_change":"2023-10-03T17:40:24.472734+0000","last_active":"2023-10-03T17:52:02.561058+0000","last_peered":"2023-10-03T17:52:02.561058+0000","last_clean":"2023-10-03T17:52:02.561058+0000","last_became_active":"2023-10-03T17:34:42.270194+0000","last_became_peered":"2023-10-03T17:34:42.270194+0000","last_unstale":"2023-10-03T17:52:02.561058+0000","last_undegraded":"2023-10-03T17:52:02.561058+0000","last_fullsized":"2023-10-03T17:52:02.561058+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'8","last_scrub_stamp":"2023-10-03T17:40:24.472573+0000","last_deep_scrub":"46'8","last_deep_scrub_stamp":"2023-10-03T17:40:24.472573+0000","last_clean_scrub_stamp":"2023-10-03T17:40:24.472573+0000","objects_scrubbed":4,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T18:28:04.735033+0000","scrub_duration":0.024990637999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":385,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":45,"num_read_kb":27,"num_write":47,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"46'99","reported_seq":242,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-03T17:52:02.449635+0000","last_change":"2023-10-03T17:40:10.452341+0000","last_active":"2023-10-03T17:52:02.449635+0000","last_peered":"2023-10-03T17:52:02.449635+0000","last_clean":"2023-10-03T17:52:02.449635+0000","last_became_active":"2023-10-03T17:34:39.502393+0000","last_became_peered":"2023-10-03T17:34:39.502393+0000","last_unstale":"2023-10-03T17:52:02.449635+0000","last_undegraded":"2023-10-03T17:52:02.449635+0000","last_fullsized":"2023-10-03T17:52:02.449635+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'99","last_scrub_stamp":"2023-10-03T17:40:10.449099+0000","last_deep_scrub":"46'99","last_deep_scrub_stamp":"2023-10-03T17:40:10.449099+0000","last_clean_scrub_stamp":"2023-10-03T17:40:10.449099+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T20:18:04.958666+0000","scrub_duration":0.043225012,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":22120902675,"num_objects":41409,"num_object_clones":36288,"num_object_copies":82818,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41409,"num_whiteouts":0,"num_read":384799,"num_read_kb":1458228,"num_write":389273,"num_write_kb":1557086,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3189731328,"data_stored":18849001510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40213,"ondisk_log_size":40213,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":5372,"num_objects":143,"num_object_clones":0,"num_object_copies":286,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":143,"num_whiteouts":0,"num_read":54670,"num_read_kb":45687,"num_write":2283,"num_write_kb":1803,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":983040,"data_stored":10744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7029,"internal_metadata":0},"log_size":11930,"ondisk_log_size":11930,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":7460,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738589,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2631152,"kb_used_data":1557928,"kb_used_omap":2891,"kb_used_meta":1070324,"kb_avail":91740688,"statfs":{"total":96636764160,"available":93942464512,"internally_reserved":0,"allocated":1595318272,"data_stored":9421563593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2960945,"internal_metadata":1096012239},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,5,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 17:52:06 2023","interfaces":[{"interface":"back","average":{"1min":0.32000000000000001,"5min":0.60399999999999998,"15min":0.47199999999999998},"min":{"1min":0.158,"5min":0.158,"15min":0.158},"max":{"1min":0.57999999999999996,"5min":14.787000000000001,"15min":14.787000000000001},"last":0.26300000000000001},{"interface":"front","average":{"1min":0.30499999999999999,"5min":0.59599999999999997,"15min":0.46700000000000003},"min":{"1min":0.186,"5min":0.185,"15min":0.17100000000000001},"max":{"1min":0.47199999999999998,"5min":14.919,"15min":14.919},"last":0.29299999999999998}]},{"osd":1,"last update":"Tue Oct 3 17:52:06 2023","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.58799999999999997,"15min":0.53200000000000003},"min":{"1min":0.23300000000000001,"5min":0.218,"15min":0.17599999999999999},"max":{"1min":3.5659999999999998,"5min":6.7539999999999996,"15min":8.1059999999999999},"last":0.23100000000000001},{"interface":"front","average":{"1min":0.34899999999999998,"5min":0.51100000000000001,"15min":0.46500000000000002},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.17699999999999999},"max":{"1min":0.62,"5min":5.194,"15min":5.194},"last":0.372}]}]},{"osd":1,"up_from":8,"seq":34359738587,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2193232,"kb_used_data":767944,"kb_used_omap":1,"kb_used_meta":1425278,"kb_avail":92178608,"statfs":{"total":96636764160,"available":94390894592,"internally_reserved":0,"allocated":786374656,"data_stored":5483919058,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":1459485126},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,0,1,2,8,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 17:52:18 2023","interfaces":[{"interface":"back","average":{"1min":0.28799999999999998,"5min":0.33600000000000002,"15min":0.33700000000000002},"min":{"1min":0.17599999999999999,"5min":0.17599999999999999,"15min":0.161},"max":{"1min":0.47099999999999997,"5min":2.7050000000000001,"15min":4.1879999999999997},"last":0.24199999999999999},{"interface":"front","average":{"1min":0.372,"5min":0.38800000000000001,"15min":0.38500000000000001},"min":{"1min":0.21299999999999999,"5min":0.191,"15min":0.16800000000000001},"max":{"1min":1.913,"5min":5.8959999999999999,"15min":5.8959999999999999},"last":0.217}]},{"osd":2,"last update":"Tue Oct 3 17:52:18 2023","interfaces":[{"interface":"back","average":{"1min":0.33900000000000002,"5min":0.433,"15min":0.40200000000000002},"min":{"1min":0.20999999999999999,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":0.68999999999999995,"5min":4.5519999999999996,"15min":4.5519999999999996},"last":0.28699999999999998},{"interface":"front","average":{"1min":0.35299999999999998,"5min":0.39300000000000002,"15min":0.377},"min":{"1min":0.25800000000000001,"5min":0.19700000000000001,"15min":0.16400000000000001},"max":{"1min":0.498,"5min":2.665,"15min":3.0699999999999998},"last":0.27700000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738587,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2288576,"kb_used_data":791288,"kb_used_omap":1,"kb_used_meta":1497278,"kb_avail":92083264,"statfs":{"total":96636764160,"available":94293262336,"internally_reserved":0,"allocated":810278912,"data_stored":3937939372,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":1533213126},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,0,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 17:52:04 2023","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.39600000000000002,"15min":0.41499999999999998},"min":{"1min":0.182,"5min":0.151,"15min":0.151},"max":{"1min":2.8340000000000001,"5min":3.9430000000000001,"15min":5.7649999999999997},"last":0.30199999999999999},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.49299999999999999,"15min":0.47999999999999998},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.16900000000000001},"max":{"1min":3.6930000000000001,"5min":3.9169999999999998,"15min":9.4770000000000003},"last":0.218}]},{"osd":2,"last update":"Tue Oct 3 17:52:04 2023","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.47199999999999998,"15min":0.51100000000000001},"min":{"1min":0.215,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":3.996,"5min":3.996,"15min":9.4149999999999991},"last":0.26700000000000002},{"interface":"front","average":{"1min":0.31900000000000001,"5min":0.36799999999999999,"15min":0.46899999999999997},"min":{"1min":0.19600000000000001,"5min":0.19600000000000001,"15min":0.189},"max":{"1min":0.64900000000000002,"5min":1.046,"15min":5.218},"last":0.252}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":3561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":5372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7029,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":809213952,"data_stored":3938209792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":785272832,"data_stored":5484199955,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1595244544,"data_stored":9426591763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-03T17:52:38.595 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-03T17:34:47.281095+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=34, tm_sec=47, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=40, tm_sec=6, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T17:52:38.595 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.0 2023-10-03T17:52:39.367 INFO:teuthology.orchestra.run.smithi111.stderr:instructing pg 3.0 on osd.1 to deep-scrub 2023-10-03T17:52:39.405 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T17:52:57.343 INFO:tasks.ceph.osd.0.smithi111.stderr:2023-10-03T17:52:57.345+0000 7ffbac289640 -1 reset not still connected to 0x5563a7ff68f0 2023-10-03T17:52:57.343 INFO:tasks.ceph.osd.0.smithi111.stderr:2023-10-03T17:52:57.345+0000 7ffbac289640 -1 reset not still connected to 0x5563a838e1a0 2023-10-03T17:52:57.343 INFO:tasks.ceph.osd.0.smithi111.stderr:2023-10-03T17:52:57.345+0000 7ffbac289640 -1 reset not still connected to 0x5563a851aea0 2023-10-03T17:52:59.406 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T17:53:00.082 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-03T17:53:00.082 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-10-03T17:53:00.128 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":646,"stamp":"2023-10-03T17:52:58.225105+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":22149973807,"num_objects":41184,"num_object_clones":35918,"num_object_copies":82368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41184,"num_whiteouts":0,"num_read":445478,"num_read_kb":1527262,"num_write":397752,"num_write_kb":1585019,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52290,"ondisk_log_size":52290,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":7300612,"kb_used_data":3165224,"kb_used_omap":2894,"kb_used_meta":4132465,"kb_avail":275814908,"statfs":{"total":289910292480,"available":282434465792,"internally_reserved":0,"allocated":3241189376,"data_stored":18733161799,"data_compressed":7460,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":2964133,"internal_metadata":4231644507},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,2,6,6,14,19],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":15110144,"num_objects":-324,"num_object_clones":-324,"num_object_copies":-648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-324,"num_whiteouts":0,"num_read":3156,"num_read_kb":12436,"num_write":3205,"num_write_kb":12820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002200"},"pg_stats":[{"pgid":"2.7","version":"97'2169","reported_seq":4696,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-03T17:52:53.920843+0000","last_change":"2023-10-03T17:40:17.407011+0000","last_active":"2023-10-03T17:52:53.920843+0000","last_peered":"2023-10-03T17:52:53.920843+0000","last_clean":"2023-10-03T17:52:53.920843+0000","last_became_active":"2023-10-03T17:34:42.276624+0000","last_became_peered":"2023-10-03T17:34:42.276624+0000","last_unstale":"2023-10-03T17:52:53.920843+0000","last_undegraded":"2023-10-03T17:52:53.920843+0000","last_fullsized":"2023-10-03T17:52:53.920843+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'673","last_scrub_stamp":"2023-10-03T17:40:17.406813+0000","last_deep_scrub":"46'673","last_deep_scrub_stamp":"2023-10-03T17:40:17.406813+0000","last_clean_scrub_stamp":"2023-10-03T17:40:17.406813+0000","objects_scrubbed":7,"log_size":2169,"log_dups_size":0,"ondisk_log_size":2169,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T21:38:35.056889+0000","scrub_duration":0.013778076,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":688,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":7673,"num_read_kb":6341,"num_write":331,"num_write_kb":257,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":903,"num_omap_keys":30,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"97'3579","reported_seq":9434,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-03T17:52:53.929096+0000","last_change":"2023-10-03T17:40:13.376519+0000","last_active":"2023-10-03T17:52:53.929096+0000","last_peered":"2023-10-03T17:52:53.929096+0000","last_clean":"2023-10-03T17:52:53.929096+0000","last_became_active":"2023-10-03T17:34:42.271201+0000","last_became_peered":"2023-10-03T17:34:42.271201+0000","last_unstale":"2023-10-03T17:52:53.929096+0000","last_undegraded":"2023-10-03T17:52:53.929096+0000","last_fullsized":"2023-10-03T17:52:53.929096+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'1129","last_scrub_stamp":"2023-10-03T17:40:13.376332+0000","last_deep_scrub":"46'1129","last_deep_scrub_stamp":"2023-10-03T17:40:13.376332+0000","last_clean_scrub_stamp":"2023-10-03T17:40:13.376332+0000","objects_scrubbed":11,"log_size":3579,"log_dups_size":0,"ondisk_log_size":3579,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T18:59:17.088585+0000","scrub_duration":0.015998675,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":623,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":15229,"num_read_kb":12999,"num_write":544,"num_write_kb":440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2912,"num_omap_keys":93,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"96'59","reported_seq":204,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-03T17:52:03.303639+0000","last_change":"2023-10-03T17:40:25.496010+0000","last_active":"2023-10-03T17:52:03.303639+0000","last_peered":"2023-10-03T17:52:03.303639+0000","last_clean":"2023-10-03T17:52:03.303639+0000","last_became_active":"2023-10-03T17:34:42.276941+0000","last_became_peered":"2023-10-03T17:34:42.276941+0000","last_unstale":"2023-10-03T17:52:03.303639+0000","last_undegraded":"2023-10-03T17:52:03.303639+0000","last_fullsized":"2023-10-03T17:52:03.303639+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'14","last_scrub_stamp":"2023-10-03T17:40:25.495827+0000","last_deep_scrub":"46'14","last_deep_scrub_stamp":"2023-10-03T17:40:25.495827+0000","last_clean_scrub_stamp":"2023-10-03T17:40:25.495827+0000","objects_scrubbed":7,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:05:44.939663+0000","scrub_duration":0.0133078,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1107,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":53,"num_read_kb":26,"num_write":52,"num_write_kb":38,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"97'2867","reported_seq":7082,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-03T17:52:53.925554+0000","last_change":"2023-10-03T17:40:26.513902+0000","last_active":"2023-10-03T17:52:53.925554+0000","last_peered":"2023-10-03T17:52:53.925554+0000","last_clean":"2023-10-03T17:52:53.925554+0000","last_became_active":"2023-10-03T17:34:42.276459+0000","last_became_peered":"2023-10-03T17:34:42.276459+0000","last_unstale":"2023-10-03T17:52:53.925554+0000","last_undegraded":"2023-10-03T17:52:53.925554+0000","last_fullsized":"2023-10-03T17:52:53.925554+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'964","last_scrub_stamp":"2023-10-03T17:40:26.513729+0000","last_deep_scrub":"46'964","last_deep_scrub_stamp":"2023-10-03T17:40:26.513729+0000","last_clean_scrub_stamp":"2023-10-03T17:40:26.513729+0000","objects_scrubbed":10,"log_size":2867,"log_dups_size":0,"ondisk_log_size":2867,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:41:33.182751+0000","scrub_duration":0.018190280999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":335,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":11172,"num_read_kb":9347,"num_write":412,"num_write_kb":333,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1731,"num_omap_keys":80,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"97'133523","reported_seq":216473,"reported_epoch":97,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T17:52:54.234684+0000","last_change":"2023-10-03T17:52:53.400862+0000","last_active":"2023-10-03T17:52:54.234684+0000","last_peered":"2023-10-03T17:52:54.234684+0000","last_clean":"2023-10-03T17:52:54.234684+0000","last_became_active":"2023-10-03T17:34:48.310701+0000","last_became_peered":"2023-10-03T17:34:48.310701+0000","last_unstale":"2023-10-03T17:52:54.234684+0000","last_undegraded":"2023-10-03T17:52:54.234684+0000","last_fullsized":"2023-10-03T17:52:54.234684+0000","mapping_epoch":16,"log_start":"93'123513","ondisk_log_start":"93'123513","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'80695","last_scrub_stamp":"2023-10-03T17:46:13.982571+0000","last_deep_scrub":"66'80695","last_deep_scrub_stamp":"2023-10-03T17:46:13.982571+0000","last_clean_scrub_stamp":"2023-10-03T17:46:13.982571+0000","objects_scrubbed":3410,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":53,"last_scrub_duration":363,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:15:57.360096+0000","scrub_duration":362.26574615599998,"objects_trimmed":717,"snaptrim_duration":49.931710838999997,"stat_sum":{"num_bytes":5708931072,"num_objects":9845,"num_object_clones":8525,"num_object_copies":19690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9845,"num_whiteouts":0,"num_read":100891,"num_read_kb":382420,"num_write":102258,"num_write_kb":409032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"54","length":"1e"},{"start":"7c","length":"8"},{"start":"8a","length":"2"},{"start":"90","length":"a"},{"start":"a4","length":"a"},{"start":"b8","length":"a"},{"start":"cc","length":"a"},{"start":"e0","length":"a"},{"start":"f4","length":"a"},{"start":"108","length":"7"}]},{"pgid":"2.2","version":"97'2965","reported_seq":6328,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-03T17:52:51.767132+0000","last_change":"2023-10-03T17:40:08.675119+0000","last_active":"2023-10-03T17:52:51.767132+0000","last_peered":"2023-10-03T17:52:51.767132+0000","last_clean":"2023-10-03T17:52:51.767132+0000","last_became_active":"2023-10-03T17:34:42.270625+0000","last_became_peered":"2023-10-03T17:34:42.270625+0000","last_unstale":"2023-10-03T17:52:51.767132+0000","last_undegraded":"2023-10-03T17:52:51.767132+0000","last_fullsized":"2023-10-03T17:52:51.767132+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'841","last_scrub_stamp":"2023-10-03T17:40:08.674910+0000","last_deep_scrub":"46'841","last_deep_scrub_stamp":"2023-10-03T17:40:08.674910+0000","last_clean_scrub_stamp":"2023-10-03T17:40:08.674910+0000","objects_scrubbed":11,"log_size":2965,"log_dups_size":0,"ondisk_log_size":2965,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T21:05:42.790614+0000","scrub_duration":0.039189555000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":808,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":10254,"num_read_kb":8491,"num_write":442,"num_write_kb":352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2564,"num_omap_keys":60,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"97'117448","reported_seq":198472,"reported_epoch":97,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T17:52:54.190719+0000","last_change":"2023-10-03T17:43:28.494717+0000","last_active":"2023-10-03T17:52:54.190719+0000","last_peered":"2023-10-03T17:52:54.190719+0000","last_clean":"2023-10-03T17:52:54.190719+0000","last_became_active":"2023-10-03T17:34:48.310555+0000","last_became_peered":"2023-10-03T17:34:48.310555+0000","last_unstale":"2023-10-03T17:52:54.190719+0000","last_undegraded":"2023-10-03T17:52:54.190719+0000","last_fullsized":"2023-10-03T17:52:54.190719+0000","mapping_epoch":16,"log_start":"93'107435","ondisk_log_start":"93'107435","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_clean_scrub_stamp":"2023-10-03T17:34:47.281095+0000","objects_scrubbed":8787,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":160,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 565s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086289699999999995,"stat_sum":{"num_bytes":5371392000,"num_objects":15778,"num_object_clones":14536,"num_object_copies":31556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15778,"num_whiteouts":0,"num_read":94862,"num_read_kb":358824,"num_write":95249,"num_write_kb":380996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"97'272","reported_seq":2781,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-03T17:52:51.807293+0000","last_change":"2023-10-03T17:40:17.641493+0000","last_active":"2023-10-03T17:52:51.807293+0000","last_peered":"2023-10-03T17:52:51.807293+0000","last_clean":"2023-10-03T17:52:51.807293+0000","last_became_active":"2023-10-03T17:34:42.278196+0000","last_became_peered":"2023-10-03T17:34:42.278196+0000","last_unstale":"2023-10-03T17:52:51.807293+0000","last_undegraded":"2023-10-03T17:52:51.807293+0000","last_fullsized":"2023-10-03T17:52:51.807293+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'30","last_scrub_stamp":"2023-10-03T17:40:17.640969+0000","last_deep_scrub":"46'30","last_deep_scrub_stamp":"2023-10-03T17:40:17.640969+0000","last_clean_scrub_stamp":"2023-10-03T17:40:17.640969+0000","objects_scrubbed":6,"log_size":272,"log_dups_size":0,"ondisk_log_size":272,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:04:02.954896+0000","scrub_duration":0.071054565,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":522,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":7642,"num_read_kb":6319,"num_write":305,"num_write_kb":242,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":903,"num_omap_keys":30,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"97'132377","reported_seq":211027,"reported_epoch":97,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T17:52:54.231997+0000","last_change":"2023-10-03T17:52:53.261712+0000","last_active":"2023-10-03T17:52:54.231997+0000","last_peered":"2023-10-03T17:52:54.231997+0000","last_clean":"2023-10-03T17:52:54.231997+0000","last_became_active":"2023-10-03T17:34:48.310912+0000","last_became_peered":"2023-10-03T17:34:48.310912+0000","last_unstale":"2023-10-03T17:52:54.231997+0000","last_undegraded":"2023-10-03T17:52:54.231997+0000","last_fullsized":"2023-10-03T17:52:54.231997+0000","mapping_epoch":16,"log_start":"93'122372","ondisk_log_start":"93'122372","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'58196","last_scrub_stamp":"2023-10-03T17:43:26.193075+0000","last_deep_scrub":"53'58196","last_deep_scrub_stamp":"2023-10-03T17:43:26.193075+0000","last_clean_scrub_stamp":"2023-10-03T17:43:26.193075+0000","objects_scrubbed":2326,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":14,"last_scrub_duration":196,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T19:30:13.881336+0000","scrub_duration":195.44739730800001,"objects_trimmed":811,"snaptrim_duration":49.795444263,"stat_sum":{"num_bytes":5501403136,"num_objects":7058,"num_object_clones":5787,"num_object_copies":14116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7058,"num_whiteouts":0,"num_read":97017,"num_read_kb":368616,"num_write":98372,"num_write_kb":393488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"54","length":"1e"},{"start":"7c","length":"8"},{"start":"8a","length":"2"},{"start":"90","length":"a"},{"start":"a4","length":"a"},{"start":"b8","length":"a"},{"start":"cc","length":"a"},{"start":"e0","length":"a"},{"start":"f4","length":"a"},{"start":"108","length":"a"},{"start":"11c","length":"a"},{"start":"130","length":"a"},{"start":"143","length":"7"},{"start":"150","length":"1"},{"start":"152","length":"1"},{"start":"154","length":"1"},{"start":"158","length":"6"}]},{"pgid":"2.0","version":"96'144","reported_seq":1090,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-03T17:52:49.798044+0000","last_change":"2023-10-03T17:40:10.708344+0000","last_active":"2023-10-03T17:52:49.798044+0000","last_peered":"2023-10-03T17:52:49.798044+0000","last_clean":"2023-10-03T17:52:49.798044+0000","last_became_active":"2023-10-03T17:34:42.272306+0000","last_became_peered":"2023-10-03T17:34:42.272306+0000","last_unstale":"2023-10-03T17:52:49.798044+0000","last_undegraded":"2023-10-03T17:52:49.798044+0000","last_fullsized":"2023-10-03T17:52:49.798044+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'20","last_scrub_stamp":"2023-10-03T17:40:10.708060+0000","last_deep_scrub":"46'20","last_deep_scrub_stamp":"2023-10-03T17:40:10.708060+0000","last_clean_scrub_stamp":"2023-10-03T17:40:10.708060+0000","objects_scrubbed":7,"log_size":144,"log_dups_size":0,"ondisk_log_size":144,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:03:09.097252+0000","scrub_duration":0.042279491000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":904,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":2602,"num_read_kb":2137,"num_write":150,"num_write_kb":113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":301,"num_omap_keys":10,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"97'132711","reported_seq":212621,"reported_epoch":97,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T17:52:54.213589+0000","last_change":"2023-10-03T17:52:52.658124+0000","last_active":"2023-10-03T17:52:54.213589+0000","last_peered":"2023-10-03T17:52:54.213589+0000","last_clean":"2023-10-03T17:52:54.213589+0000","last_became_active":"2023-10-03T17:34:48.310612+0000","last_became_peered":"2023-10-03T17:34:48.310612+0000","last_unstale":"2023-10-03T17:52:54.213589+0000","last_undegraded":"2023-10-03T17:52:54.213589+0000","last_fullsized":"2023-10-03T17:52:54.213589+0000","mapping_epoch":16,"log_start":"93'122654","ondisk_log_start":"93'122654","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'75807","last_scrub_stamp":"2023-10-03T17:45:43.833310+0000","last_deep_scrub":"61'75807","last_deep_scrub_stamp":"2023-10-03T17:45:43.833310+0000","last_clean_scrub_stamp":"2023-10-03T17:45:43.833310+0000","objects_scrubbed":3607,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":33,"last_scrub_duration":315,"scrub_schedule":"queued for deep scrub","scrub_duration":314.26374841099999,"objects_trimmed":807,"snaptrim_duration":49.190057830000001,"stat_sum":{"num_bytes":5567651859,"num_objects":8358,"num_object_clones":7070,"num_object_copies":16716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8358,"num_whiteouts":0,"num_read":97932,"num_read_kb":371624,"num_write":99453,"num_write_kb":397806,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"54","length":"1e"},{"start":"7c","length":"8"},{"start":"8a","length":"2"},{"start":"90","length":"a"},{"start":"a4","length":"a"},{"start":"b8","length":"a"},{"start":"cc","length":"a"},{"start":"e0","length":"a"},{"start":"f4","length":"a"},{"start":"108","length":"a"},{"start":"11c","length":"a"},{"start":"130","length":"7"}]},{"pgid":"2.3","version":"96'51","reported_seq":194,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-03T17:52:02.561058+0000","last_change":"2023-10-03T17:40:24.472734+0000","last_active":"2023-10-03T17:52:02.561058+0000","last_peered":"2023-10-03T17:52:02.561058+0000","last_clean":"2023-10-03T17:52:02.561058+0000","last_became_active":"2023-10-03T17:34:42.270194+0000","last_became_peered":"2023-10-03T17:34:42.270194+0000","last_unstale":"2023-10-03T17:52:02.561058+0000","last_undegraded":"2023-10-03T17:52:02.561058+0000","last_fullsized":"2023-10-03T17:52:02.561058+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'8","last_scrub_stamp":"2023-10-03T17:40:24.472573+0000","last_deep_scrub":"46'8","last_deep_scrub_stamp":"2023-10-03T17:40:24.472573+0000","last_clean_scrub_stamp":"2023-10-03T17:40:24.472573+0000","objects_scrubbed":4,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T18:28:04.735033+0000","scrub_duration":0.024990637999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":385,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":45,"num_read_kb":27,"num_write":47,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"46'99","reported_seq":243,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-03T17:52:03.464689+0000","last_change":"2023-10-03T17:40:10.452341+0000","last_active":"2023-10-03T17:52:03.464689+0000","last_peered":"2023-10-03T17:52:03.464689+0000","last_clean":"2023-10-03T17:52:03.464689+0000","last_became_active":"2023-10-03T17:34:39.502393+0000","last_became_peered":"2023-10-03T17:34:39.502393+0000","last_unstale":"2023-10-03T17:52:03.464689+0000","last_undegraded":"2023-10-03T17:52:03.464689+0000","last_fullsized":"2023-10-03T17:52:03.464689+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'99","last_scrub_stamp":"2023-10-03T17:40:10.449099+0000","last_deep_scrub":"46'99","last_deep_scrub_stamp":"2023-10-03T17:40:10.449099+0000","last_clean_scrub_stamp":"2023-10-03T17:40:10.449099+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T20:18:04.958666+0000","scrub_duration":0.043225012,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":22149378067,"num_objects":41039,"num_object_clones":35918,"num_object_copies":82078,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41039,"num_whiteouts":0,"num_read":390702,"num_read_kb":1481484,"num_write":395332,"num_write_kb":1581322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3239297024,"data_stored":18717937702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40085,"ondisk_log_size":40085,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":5372,"num_objects":143,"num_object_clones":0,"num_object_copies":286,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":143,"num_whiteouts":0,"num_read":54670,"num_read_kb":45687,"num_write":2283,"num_write_kb":1803,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":983040,"data_stored":10744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7029,"internal_metadata":0},"log_size":12106,"ondisk_log_size":12106,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":7460,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738593,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2683080,"kb_used_data":1582076,"kb_used_omap":2891,"kb_used_meta":1098100,"kb_avail":91688760,"statfs":{"total":96636764160,"available":93889290240,"internally_reserved":0,"allocated":1620045824,"data_stored":9363355337,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2960945,"internal_metadata":1124454863},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 17:52:06 2023","interfaces":[{"interface":"back","average":{"1min":0.32000000000000001,"5min":0.60399999999999998,"15min":0.47199999999999998},"min":{"1min":0.158,"5min":0.158,"15min":0.158},"max":{"1min":0.57999999999999996,"5min":14.787000000000001,"15min":14.787000000000001},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.30499999999999999,"5min":0.59599999999999997,"15min":0.46700000000000003},"min":{"1min":0.186,"5min":0.185,"15min":0.17100000000000001},"max":{"1min":0.47199999999999998,"5min":14.919,"15min":14.919},"last":0.34999999999999998}]},{"osd":1,"last update":"Tue Oct 3 17:52:06 2023","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.58799999999999997,"15min":0.53200000000000003},"min":{"1min":0.23300000000000001,"5min":0.218,"15min":0.17599999999999999},"max":{"1min":3.5659999999999998,"5min":6.7539999999999996,"15min":8.1059999999999999},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.34899999999999998,"5min":0.51100000000000001,"15min":0.46500000000000002},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.17699999999999999},"max":{"1min":0.62,"5min":5.194,"15min":5.194},"last":0.313}]}]},{"osd":1,"up_from":8,"seq":34359738591,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2278240,"kb_used_data":779800,"kb_used_omap":1,"kb_used_meta":1498430,"kb_avail":92093600,"statfs":{"total":96636764160,"available":94303846400,"internally_reserved":0,"allocated":798515200,"data_stored":5505267410,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":1534392774},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,1,3,0,10],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 17:52:18 2023","interfaces":[{"interface":"back","average":{"1min":0.28799999999999998,"5min":0.33600000000000002,"15min":0.33700000000000002},"min":{"1min":0.17599999999999999,"5min":0.17599999999999999,"15min":0.161},"max":{"1min":0.47099999999999997,"5min":2.7050000000000001,"15min":4.1879999999999997},"last":0.27500000000000002},{"interface":"front","average":{"1min":0.372,"5min":0.38800000000000001,"15min":0.38500000000000001},"min":{"1min":0.21299999999999999,"5min":0.191,"15min":0.16800000000000001},"max":{"1min":1.913,"5min":5.8959999999999999,"15min":5.8959999999999999},"last":0.28999999999999998}]},{"osd":2,"last update":"Tue Oct 3 17:52:18 2023","interfaces":[{"interface":"back","average":{"1min":0.33900000000000002,"5min":0.433,"15min":0.40200000000000002},"min":{"1min":0.20999999999999999,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":0.68999999999999995,"5min":4.5519999999999996,"15min":4.5519999999999996},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.35299999999999998,"5min":0.39300000000000002,"15min":0.377},"min":{"1min":0.25800000000000001,"5min":0.19700000000000001,"15min":0.16400000000000001},"max":{"1min":0.498,"5min":2.665,"15min":3.0699999999999998},"last":0.24099999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738591,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2339292,"kb_used_data":803348,"kb_used_omap":1,"kb_used_meta":1535934,"kb_avail":92032548,"statfs":{"total":96636764160,"available":94241329152,"internally_reserved":0,"allocated":822628352,"data_stored":3864539052,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":1572796870},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,4,4,19],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 17:52:04 2023","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.39600000000000002,"15min":0.41499999999999998},"min":{"1min":0.182,"5min":0.151,"15min":0.151},"max":{"1min":2.8340000000000001,"5min":3.9430000000000001,"15min":5.7649999999999997},"last":0.20399999999999999},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.49299999999999999,"15min":0.47999999999999998},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.16900000000000001},"max":{"1min":3.6930000000000001,"5min":3.9169999999999998,"15min":9.4770000000000003},"last":0.27000000000000002}]},{"osd":2,"last update":"Tue Oct 3 17:52:04 2023","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.47199999999999998,"15min":0.51100000000000001},"min":{"1min":0.215,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":3.996,"5min":3.996,"15min":9.4149999999999991},"last":0.38},{"interface":"front","average":{"1min":0.31900000000000001,"5min":0.36799999999999999,"15min":0.46899999999999997},"min":{"1min":0.19600000000000001,"5min":0.19600000000000001,"15min":0.189},"max":{"1min":0.64900000000000002,"5min":1.046,"15min":5.218},"last":0.29999999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":3561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":5372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7029,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":821784576,"data_stored":3861917696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":797499392,"data_stored":5501468691,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1620013056,"data_stored":9354551315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-03T17:53:00.130 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-03T17:34:47.281095+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=34, tm_sec=47, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=40, tm_sec=6, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T17:53:00.130 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T17:53:11.678 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:53:11.680+0000 7f09e4851640 -1 reset not still connected to 0x557d75f2b2b0 2023-10-03T17:53:11.679 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:53:11.680+0000 7f09e4851640 -1 reset not still connected to 0x557d76253d40 2023-10-03T17:53:11.679 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:53:11.680+0000 7f09e4851640 -1 reset not still connected to 0x557d76621040 2023-10-03T17:53:11.679 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:53:11.680+0000 7f09e4851640 -1 reset not still connected to 0x557d7664e750 2023-10-03T17:53:11.679 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:53:11.680+0000 7f09e4851640 -1 reset not still connected to 0x557d7664ea90 2023-10-03T17:53:11.681 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:53:11.684+0000 7f09e4851640 -1 reset not still connected to 0x557d7664f380 2023-10-03T17:53:11.681 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:53:11.684+0000 7f09e4851640 -1 reset not still connected to 0x557d7664f930 2023-10-03T17:53:11.681 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:53:11.684+0000 7f09e4851640 -1 reset not still connected to 0x557d76681ba0 2023-10-03T17:53:11.681 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:53:11.684+0000 7f09e4851640 -1 reset not still connected to 0x557d766a40d0 2023-10-03T17:53:11.681 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:53:11.684+0000 7f09e4851640 -1 reset not still connected to 0x557d766b56c0 2023-10-03T17:53:11.681 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:53:11.684+0000 7f09e4851640 -1 reset not still connected to 0x557d76eca270 2023-10-03T17:53:11.682 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-10-03T17:53:11.684+0000 7f09e4851640 -1 reset not still connected to 0x557d76f93ba0 2023-10-03T17:53:20.131 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T17:53:20.657 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-03T17:53:20.658 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-10-03T17:53:20.685 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":663,"stamp":"2023-10-03T17:53:20.232608+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":22183368495,"num_objects":42949,"num_object_clones":37683,"num_object_copies":85898,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42949,"num_whiteouts":0,"num_read":455242,"num_read_kb":1556486,"num_write":404631,"num_write_kb":1611815,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52706,"ondisk_log_size":52706,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":7589128,"kb_used_data":3219504,"kb_used_omap":2894,"kb_used_meta":4366705,"kb_avail":275526392,"statfs":{"total":289910292480,"available":282139025408,"internally_reserved":0,"allocated":3296772096,"data_stored":20004060633,"data_compressed":7460,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":2964133,"internal_metadata":4471506267},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,2,8,12,22,31,7],"upper_bound":512},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":17547264,"num_objects":1621,"num_object_clones":1621,"num_object_copies":3242,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1621,"num_whiteouts":0,"num_read":3833,"num_read_kb":14894,"num_write":3654,"num_write_kb":14575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005834"},"pg_stats":[{"pgid":"2.7","version":"103'2233","reported_seq":4894,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-10-03T17:53:18.989081+0000","last_change":"2023-10-03T17:40:17.407011+0000","last_active":"2023-10-03T17:53:18.989081+0000","last_peered":"2023-10-03T17:53:18.989081+0000","last_clean":"2023-10-03T17:53:18.989081+0000","last_became_active":"2023-10-03T17:34:42.276624+0000","last_became_peered":"2023-10-03T17:34:42.276624+0000","last_unstale":"2023-10-03T17:53:18.989081+0000","last_undegraded":"2023-10-03T17:53:18.989081+0000","last_fullsized":"2023-10-03T17:53:18.989081+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'673","last_scrub_stamp":"2023-10-03T17:40:17.406813+0000","last_deep_scrub":"46'673","last_deep_scrub_stamp":"2023-10-03T17:40:17.406813+0000","last_clean_scrub_stamp":"2023-10-03T17:40:17.406813+0000","objects_scrubbed":7,"log_size":2233,"log_dups_size":0,"ondisk_log_size":2233,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T21:38:35.056889+0000","scrub_duration":0.013778076,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":639,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":8113,"num_read_kb":6709,"num_write":363,"num_write_kb":280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":903,"num_omap_keys":30,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"103'3680","reported_seq":9820,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-10-03T17:53:18.996927+0000","last_change":"2023-10-03T17:40:13.376519+0000","last_active":"2023-10-03T17:53:18.996927+0000","last_peered":"2023-10-03T17:53:18.996927+0000","last_clean":"2023-10-03T17:53:18.996927+0000","last_became_active":"2023-10-03T17:34:42.271201+0000","last_became_peered":"2023-10-03T17:34:42.271201+0000","last_unstale":"2023-10-03T17:53:18.996927+0000","last_undegraded":"2023-10-03T17:53:18.996927+0000","last_fullsized":"2023-10-03T17:53:18.996927+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'1129","last_scrub_stamp":"2023-10-03T17:40:13.376332+0000","last_deep_scrub":"46'1129","last_deep_scrub_stamp":"2023-10-03T17:40:13.376332+0000","last_clean_scrub_stamp":"2023-10-03T17:40:13.376332+0000","objects_scrubbed":11,"log_size":3680,"log_dups_size":0,"ondisk_log_size":3680,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T18:59:17.088585+0000","scrub_duration":0.015998675,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":673,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":16029,"num_read_kb":13679,"num_write":592,"num_write_kb":477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2912,"num_omap_keys":93,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"102'66","reported_seq":221,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-03T17:53:06.760132+0000","last_change":"2023-10-03T17:40:25.496010+0000","last_active":"2023-10-03T17:53:06.760132+0000","last_peered":"2023-10-03T17:53:06.760132+0000","last_clean":"2023-10-03T17:53:06.760132+0000","last_became_active":"2023-10-03T17:34:42.276941+0000","last_became_peered":"2023-10-03T17:34:42.276941+0000","last_unstale":"2023-10-03T17:53:06.760132+0000","last_undegraded":"2023-10-03T17:53:06.760132+0000","last_fullsized":"2023-10-03T17:53:06.760132+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'14","last_scrub_stamp":"2023-10-03T17:40:25.495827+0000","last_deep_scrub":"46'14","last_deep_scrub_stamp":"2023-10-03T17:40:25.495827+0000","last_clean_scrub_stamp":"2023-10-03T17:40:25.495827+0000","objects_scrubbed":7,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:05:44.939663+0000","scrub_duration":0.0133078,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1057,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":57,"num_read_kb":30,"num_write":59,"num_write_kb":41,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"103'2950","reported_seq":7377,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-10-03T17:53:18.992199+0000","last_change":"2023-10-03T17:40:26.513902+0000","last_active":"2023-10-03T17:53:18.992199+0000","last_peered":"2023-10-03T17:53:18.992199+0000","last_clean":"2023-10-03T17:53:18.992199+0000","last_became_active":"2023-10-03T17:34:42.276459+0000","last_became_peered":"2023-10-03T17:34:42.276459+0000","last_unstale":"2023-10-03T17:53:18.992199+0000","last_undegraded":"2023-10-03T17:53:18.992199+0000","last_fullsized":"2023-10-03T17:53:18.992199+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'964","last_scrub_stamp":"2023-10-03T17:40:26.513729+0000","last_deep_scrub":"46'964","last_deep_scrub_stamp":"2023-10-03T17:40:26.513729+0000","last_clean_scrub_stamp":"2023-10-03T17:40:26.513729+0000","objects_scrubbed":10,"log_size":2950,"log_dups_size":0,"ondisk_log_size":2950,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:41:33.182751+0000","scrub_duration":0.018190280999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":333,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":11794,"num_read_kb":9873,"num_write":452,"num_write_kb":363,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1731,"num_omap_keys":80,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"103'136664","reported_seq":220489,"reported_epoch":103,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T17:53:19.523757+0000","last_change":"2023-10-03T17:53:16.987752+0000","last_active":"2023-10-03T17:53:19.523757+0000","last_peered":"2023-10-03T17:53:19.523757+0000","last_clean":"2023-10-03T17:53:19.523757+0000","last_became_active":"2023-10-03T17:34:48.310701+0000","last_became_peered":"2023-10-03T17:34:48.310701+0000","last_unstale":"2023-10-03T17:53:19.523757+0000","last_undegraded":"2023-10-03T17:53:19.523757+0000","last_fullsized":"2023-10-03T17:53:19.523757+0000","mapping_epoch":16,"log_start":"93'126626","ondisk_log_start":"93'126626","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'80695","last_scrub_stamp":"2023-10-03T17:46:13.982571+0000","last_deep_scrub":"66'80695","last_deep_scrub_stamp":"2023-10-03T17:46:13.982571+0000","last_clean_scrub_stamp":"2023-10-03T17:46:13.982571+0000","objects_scrubbed":3410,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":68,"last_scrub_duration":363,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T01:53:40.177844+0000","scrub_duration":362.26574615599998,"objects_trimmed":140,"snaptrim_duration":9.4876263220000006,"stat_sum":{"num_bytes":5717815296,"num_objects":10277,"num_object_clones":8957,"num_object_copies":20554,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10277,"num_whiteouts":0,"num_read":102669,"num_read_kb":389464,"num_write":103968,"num_write_kb":415872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"54","length":"1e"},{"start":"7c","length":"8"},{"start":"8a","length":"2"},{"start":"90","length":"a"},{"start":"a4","length":"a"},{"start":"b8","length":"a"},{"start":"cc","length":"a"},{"start":"e0","length":"a"},{"start":"f4","length":"a"},{"start":"108","length":"a"},{"start":"11c","length":"2"}]},{"pgid":"2.2","version":"103'3038","reported_seq":6563,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-10-03T17:53:18.635192+0000","last_change":"2023-10-03T17:40:08.675119+0000","last_active":"2023-10-03T17:53:18.635192+0000","last_peered":"2023-10-03T17:53:18.635192+0000","last_clean":"2023-10-03T17:53:18.635192+0000","last_became_active":"2023-10-03T17:34:42.270625+0000","last_became_peered":"2023-10-03T17:34:42.270625+0000","last_unstale":"2023-10-03T17:53:18.635192+0000","last_undegraded":"2023-10-03T17:53:18.635192+0000","last_fullsized":"2023-10-03T17:53:18.635192+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'841","last_scrub_stamp":"2023-10-03T17:40:08.674910+0000","last_deep_scrub":"46'841","last_deep_scrub_stamp":"2023-10-03T17:40:08.674910+0000","last_clean_scrub_stamp":"2023-10-03T17:40:08.674910+0000","objects_scrubbed":11,"log_size":3038,"log_dups_size":0,"ondisk_log_size":3038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T21:05:42.790614+0000","scrub_duration":0.039189555000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":857,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":10828,"num_read_kb":8969,"num_write":483,"num_write_kb":383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2564,"num_omap_keys":60,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"103'120107","reported_seq":202042,"reported_epoch":103,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T17:53:19.628648+0000","last_change":"2023-10-03T17:43:28.494717+0000","last_active":"2023-10-03T17:53:19.628648+0000","last_peered":"2023-10-03T17:53:19.628648+0000","last_clean":"2023-10-03T17:53:19.628648+0000","last_became_active":"2023-10-03T17:34:48.310555+0000","last_became_peered":"2023-10-03T17:34:48.310555+0000","last_unstale":"2023-10-03T17:53:19.628648+0000","last_undegraded":"2023-10-03T17:53:19.628648+0000","last_fullsized":"2023-10-03T17:53:19.628648+0000","mapping_epoch":16,"log_start":"93'110038","ondisk_log_start":"93'110038","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:34:47.281095+0000","last_clean_scrub_stamp":"2023-10-03T17:34:47.281095+0000","objects_scrubbed":9116,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":180,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 591s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086289699999999995,"stat_sum":{"num_bytes":5379362816,"num_objects":16505,"num_object_clones":15263,"num_object_copies":33010,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16505,"num_whiteouts":0,"num_read":96468,"num_read_kb":365180,"num_write":96878,"num_write_kb":387512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"103'295","reported_seq":2930,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-10-03T17:53:17.369776+0000","last_change":"2023-10-03T17:40:17.641493+0000","last_active":"2023-10-03T17:53:17.369776+0000","last_peered":"2023-10-03T17:53:17.369776+0000","last_clean":"2023-10-03T17:53:17.369776+0000","last_became_active":"2023-10-03T17:34:42.278196+0000","last_became_peered":"2023-10-03T17:34:42.278196+0000","last_unstale":"2023-10-03T17:53:17.369776+0000","last_undegraded":"2023-10-03T17:53:17.369776+0000","last_fullsized":"2023-10-03T17:53:17.369776+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'30","last_scrub_stamp":"2023-10-03T17:40:17.640969+0000","last_deep_scrub":"46'30","last_deep_scrub_stamp":"2023-10-03T17:40:17.640969+0000","last_clean_scrub_stamp":"2023-10-03T17:40:17.640969+0000","objects_scrubbed":6,"log_size":295,"log_dups_size":0,"ondisk_log_size":295,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:04:02.954896+0000","scrub_duration":0.071054565,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":471,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":8074,"num_read_kb":6679,"num_write":337,"num_write_kb":265,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":903,"num_omap_keys":30,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"103'135596","reported_seq":215011,"reported_epoch":103,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T17:53:19.509631+0000","last_change":"2023-10-03T17:53:17.177497+0000","last_active":"2023-10-03T17:53:19.509631+0000","last_peered":"2023-10-03T17:53:19.509631+0000","last_clean":"2023-10-03T17:53:19.509631+0000","last_became_active":"2023-10-03T17:34:48.310912+0000","last_became_peered":"2023-10-03T17:34:48.310912+0000","last_unstale":"2023-10-03T17:53:19.509631+0000","last_undegraded":"2023-10-03T17:53:19.509631+0000","last_fullsized":"2023-10-03T17:53:19.509631+0000","mapping_epoch":16,"log_start":"93'125580","ondisk_log_start":"93'125580","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'58196","last_scrub_stamp":"2023-10-03T17:43:26.193075+0000","last_deep_scrub":"53'58196","last_deep_scrub_stamp":"2023-10-03T17:43:26.193075+0000","last_clean_scrub_stamp":"2023-10-03T17:43:26.193075+0000","objects_scrubbed":2326,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":27,"last_scrub_duration":196,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T04:20:48.866934+0000","scrub_duration":195.44739730800001,"objects_trimmed":192,"snaptrim_duration":9.7155392190000001,"stat_sum":{"num_bytes":5509058560,"num_objects":7335,"num_object_clones":6064,"num_object_copies":14670,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7335,"num_whiteouts":0,"num_read":98660,"num_read_kb":375100,"num_write":100013,"num_write_kb":400052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"54","length":"1e"},{"start":"7c","length":"8"},{"start":"8a","length":"2"},{"start":"90","length":"a"},{"start":"a4","length":"a"},{"start":"b8","length":"a"},{"start":"cc","length":"a"},{"start":"e0","length":"a"},{"start":"f4","length":"a"},{"start":"108","length":"a"},{"start":"11c","length":"a"},{"start":"130","length":"a"},{"start":"143","length":"7"},{"start":"150","length":"1"},{"start":"152","length":"1"},{"start":"154","length":"1"},{"start":"158","length":"a"},{"start":"16c","length":"3"}]},{"pgid":"2.0","version":"102'157","reported_seq":1152,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-10-03T17:53:12.299128+0000","last_change":"2023-10-03T17:40:10.708344+0000","last_active":"2023-10-03T17:53:12.299128+0000","last_peered":"2023-10-03T17:53:12.299128+0000","last_clean":"2023-10-03T17:53:12.299128+0000","last_became_active":"2023-10-03T17:34:42.272306+0000","last_became_peered":"2023-10-03T17:34:42.272306+0000","last_unstale":"2023-10-03T17:53:12.299128+0000","last_undegraded":"2023-10-03T17:53:12.299128+0000","last_fullsized":"2023-10-03T17:53:12.299128+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'20","last_scrub_stamp":"2023-10-03T17:40:10.708060+0000","last_deep_scrub":"46'20","last_deep_scrub_stamp":"2023-10-03T17:40:10.708060+0000","last_clean_scrub_stamp":"2023-10-03T17:40:10.708060+0000","objects_scrubbed":7,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:03:09.097252+0000","scrub_duration":0.042279491000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":755,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":2750,"num_read_kb":2261,"num_write":166,"num_write_kb":122,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":301,"num_omap_keys":10,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"103'135886","reported_seq":216644,"reported_epoch":103,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T17:53:19.639696+0000","last_change":"2023-10-03T17:53:19.181806+0000","last_active":"2023-10-03T17:53:19.639696+0000","last_peered":"2023-10-03T17:53:19.639696+0000","last_clean":"2023-10-03T17:53:19.639696+0000","last_became_active":"2023-10-03T17:34:48.310612+0000","last_became_peered":"2023-10-03T17:34:48.310612+0000","last_unstale":"2023-10-03T17:53:19.639696+0000","last_undegraded":"2023-10-03T17:53:19.639696+0000","last_fullsized":"2023-10-03T17:53:19.639696+0000","mapping_epoch":16,"log_start":"93'125876","ondisk_log_start":"93'125876","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'75807","last_scrub_stamp":"2023-10-03T17:45:43.833310+0000","last_deep_scrub":"61'75807","last_deep_scrub_stamp":"2023-10-03T17:45:43.833310+0000","last_clean_scrub_stamp":"2023-10-03T17:45:43.833310+0000","objects_scrubbed":3607,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":46,"last_scrub_duration":315,"scrub_schedule":"queued for deep scrub","scrub_duration":314.26374841099999,"objects_trimmed":170,"snaptrim_duration":11.726219292,"stat_sum":{"num_bytes":5576536083,"num_objects":8687,"num_object_clones":7399,"num_object_copies":17374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8687,"num_whiteouts":0,"num_read":99649,"num_read_kb":378424,"num_write":101132,"num_write_kb":404522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"54","length":"1e"},{"start":"7c","length":"8"},{"start":"8a","length":"2"},{"start":"90","length":"a"},{"start":"a4","length":"a"},{"start":"b8","length":"a"},{"start":"cc","length":"a"},{"start":"e0","length":"a"},{"start":"f4","length":"a"},{"start":"108","length":"a"},{"start":"11c","length":"a"},{"start":"130","length":"a"},{"start":"143","length":"4"}]},{"pgid":"2.3","version":"100'55","reported_seq":204,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-03T17:53:04.444214+0000","last_change":"2023-10-03T17:40:24.472734+0000","last_active":"2023-10-03T17:53:04.444214+0000","last_peered":"2023-10-03T17:53:04.444214+0000","last_clean":"2023-10-03T17:53:04.444214+0000","last_became_active":"2023-10-03T17:34:42.270194+0000","last_became_peered":"2023-10-03T17:34:42.270194+0000","last_unstale":"2023-10-03T17:53:04.444214+0000","last_undegraded":"2023-10-03T17:53:04.444214+0000","last_fullsized":"2023-10-03T17:53:04.444214+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'8","last_scrub_stamp":"2023-10-03T17:40:24.472573+0000","last_deep_scrub":"46'8","last_deep_scrub_stamp":"2023-10-03T17:40:24.472573+0000","last_clean_scrub_stamp":"2023-10-03T17:40:24.472573+0000","objects_scrubbed":4,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T18:28:04.735033+0000","scrub_duration":0.024990637999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":587,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":45,"num_read_kb":27,"num_write":51,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"46'99","reported_seq":249,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-03T17:53:04.444151+0000","last_change":"2023-10-03T17:40:10.452341+0000","last_active":"2023-10-03T17:53:04.444151+0000","last_peered":"2023-10-03T17:53:04.444151+0000","last_clean":"2023-10-03T17:53:04.444151+0000","last_became_active":"2023-10-03T17:34:39.502393+0000","last_became_peered":"2023-10-03T17:34:39.502393+0000","last_unstale":"2023-10-03T17:53:04.444151+0000","last_undegraded":"2023-10-03T17:53:04.444151+0000","last_fullsized":"2023-10-03T17:53:04.444151+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'99","last_scrub_stamp":"2023-10-03T17:40:10.449099+0000","last_deep_scrub":"46'99","last_deep_scrub_stamp":"2023-10-03T17:40:10.449099+0000","last_clean_scrub_stamp":"2023-10-03T17:40:10.449099+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T20:18:04.958666+0000","scrub_duration":0.043225012,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":22182772755,"num_objects":42804,"num_object_clones":37683,"num_object_copies":85608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42804,"num_whiteouts":0,"num_read":397446,"num_read_kb":1508168,"num_write":401991,"num_write_kb":1607958,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3293663232,"data_stored":20011634726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40133,"ondisk_log_size":40133,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":5372,"num_objects":143,"num_object_clones":0,"num_object_copies":286,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":143,"num_whiteouts":0,"num_read":57690,"num_read_kb":48227,"num_write":2503,"num_write_kb":1963,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":983040,"data_stored":10744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7029,"internal_metadata":0},"log_size":12474,"ondisk_log_size":12474,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":7460,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738598,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2854420,"kb_used_data":1609420,"kb_used_omap":2891,"kb_used_meta":1242100,"kb_avail":91517420,"statfs":{"total":96636764160,"available":93713838080,"internally_reserved":0,"allocated":1648046080,"data_stored":10011180113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2960945,"internal_metadata":1271910863},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,3,7,13,16,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 17:53:08 2023","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.52300000000000002,"15min":0.46400000000000002},"min":{"1min":0.187,"5min":0.158,"15min":0.158},"max":{"1min":1.161,"5min":14.787000000000001,"15min":14.787000000000001},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.41899999999999998,"5min":0.57699999999999996,"15min":0.47299999999999998},"min":{"1min":0.20000000000000001,"5min":0.185,"15min":0.17100000000000001},"max":{"1min":2.9300000000000002,"5min":14.919,"15min":14.919},"last":0.24199999999999999}]},{"osd":1,"last update":"Tue Oct 3 17:53:08 2023","interfaces":[{"interface":"back","average":{"1min":0.36099999999999999,"5min":0.52700000000000002,"15min":0.52500000000000002},"min":{"1min":0.214,"5min":0.214,"15min":0.17599999999999999},"max":{"1min":1.248,"5min":6.7539999999999996,"15min":8.1059999999999999},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.54100000000000004,"15min":0.48199999999999998},"min":{"1min":0.215,"5min":0.21099999999999999,"15min":0.17699999999999999},"max":{"1min":3.9350000000000001,"5min":5.194,"15min":5.194},"last":0.40600000000000003}]}]},{"osd":1,"up_from":8,"seq":34359738596,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2327704,"kb_used_data":793168,"kb_used_omap":1,"kb_used_meta":1534526,"kb_avail":92044136,"statfs":{"total":96636764160,"available":94253195264,"internally_reserved":0,"allocated":812204032,"data_stored":5859863640,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":1571355078},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,0,3,2,5,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 17:52:18 2023","interfaces":[{"interface":"back","average":{"1min":0.28799999999999998,"5min":0.33600000000000002,"15min":0.33700000000000002},"min":{"1min":0.17599999999999999,"5min":0.17599999999999999,"15min":0.161},"max":{"1min":0.47099999999999997,"5min":2.7050000000000001,"15min":4.1879999999999997},"last":0.27500000000000002},{"interface":"front","average":{"1min":0.372,"5min":0.38800000000000001,"15min":0.38500000000000001},"min":{"1min":0.21299999999999999,"5min":0.191,"15min":0.16800000000000001},"max":{"1min":1.913,"5min":5.8959999999999999,"15min":5.8959999999999999},"last":0.22}]},{"osd":2,"last update":"Tue Oct 3 17:52:18 2023","interfaces":[{"interface":"back","average":{"1min":0.33900000000000002,"5min":0.433,"15min":0.40200000000000002},"min":{"1min":0.20999999999999999,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":0.68999999999999995,"5min":4.5519999999999996,"15min":4.5519999999999996},"last":1.7789999999999999},{"interface":"front","average":{"1min":0.35299999999999998,"5min":0.39300000000000002,"15min":0.377},"min":{"1min":0.25800000000000001,"5min":0.19700000000000001,"15min":0.16400000000000001},"max":{"1min":0.498,"5min":2.665,"15min":3.0699999999999998},"last":2.2130000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738596,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2407004,"kb_used_data":816916,"kb_used_omap":1,"kb_used_meta":1590078,"kb_avail":91964836,"statfs":{"total":96636764160,"available":94171992064,"internally_reserved":0,"allocated":836521984,"data_stored":4133016880,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":1628240326},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,0,5,2,7,10],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 17:53:05 2023","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.437,"15min":0.42799999999999999},"min":{"1min":0.184,"5min":0.151,"15min":0.151},"max":{"1min":3.1139999999999999,"5min":3.9430000000000001,"15min":5.7649999999999997},"last":0.20399999999999999},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.54100000000000004,"15min":0.48899999999999999},"min":{"1min":0.218,"5min":0.19500000000000001,"15min":0.16900000000000001},"max":{"1min":3.0670000000000002,"5min":3.9169999999999998,"15min":9.4770000000000003},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Oct 3 17:53:05 2023","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.49299999999999999,"15min":0.52000000000000002},"min":{"1min":0.218,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":2.3050000000000002,"5min":3.996,"15min":9.4149999999999991},"last":0.24299999999999999},{"interface":"front","average":{"1min":0.36399999999999999,"5min":0.36699999999999999,"15min":0.46899999999999997},"min":{"1min":0.20899999999999999,"5min":0.19600000000000001,"15min":0.189},"max":{"1min":1.179,"5min":1.179,"15min":5.218},"last":0.255}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":3559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":5372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1813,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7029,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":835530752,"data_stored":4134432768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":811044864,"data_stored":5865136147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1647087616,"data_stored":10012065811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-03T17:53:20.687 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-03T17:34:47.281095+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=34, tm_sec=47, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=40, tm_sec=6, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T17:53:20.688 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T17:53:24.662 INFO:tasks.ceph.osd.0.smithi111.stderr:2023-10-03T17:53:24.664+0000 7ffbac289640 -1 reset not still connected to 0x5563a7ff65b0 2023-10-03T17:53:24.663 INFO:tasks.ceph.osd.0.smithi111.stderr:2023-10-03T17:53:24.664+0000 7ffbac289640 -1 reset not still connected to 0x5563a7ff68f0 2023-10-03T17:53:24.663 INFO:tasks.ceph.osd.0.smithi111.stderr:2023-10-03T17:53:24.664+0000 7ffbac289640 -1 reset not still connected to 0x5563a838e1a0 2023-10-03T17:53:24.663 INFO:tasks.ceph.osd.0.smithi111.stderr:2023-10-03T17:53:24.664+0000 7ffbac289640 -1 reset not still connected to 0x5563a851aea0 2023-10-03T17:53:30.011 INFO:tasks.ceph.mgr.x.smithi111.stderr:2023-10-03T17:53:30.012+0000 7fe48bb97640 -1 calc_pg_upmaps abort due to max <= 0 2023-10-03T17:53:40.688 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T17:53:41.258 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-03T17:53:41.259 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-10-03T17:53:41.300 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":673,"stamp":"2023-10-03T17:53:40.236192+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":22209967919,"num_objects":43342,"num_object_clones":38076,"num_object_copies":86684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43342,"num_whiteouts":0,"num_read":462396,"num_read_kb":1580313,"num_write":410295,"num_write_kb":1634471,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52804,"ondisk_log_size":52804,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":7784940,"kb_used_data":3258452,"kb_used_omap":2894,"kb_used_meta":4523569,"kb_avail":275330580,"statfs":{"total":289910292480,"available":281938513920,"internally_reserved":0,"allocated":3336654848,"data_stored":20417949145,"data_compressed":7460,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":2964133,"internal_metadata":4632135003},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,3,1,6,14,17,18],"upper_bound":256},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":20250624,"num_objects":109,"num_object_clones":109,"num_object_copies":218,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":109,"num_whiteouts":0,"num_read":4404,"num_read_kb":17400,"num_write":4339,"num_write_kb":17356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002220"},"pg_stats":[{"pgid":"2.7","version":"103'2254","reported_seq":4960,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-10-03T17:53:38.413061+0000","last_change":"2023-10-03T17:40:17.407011+0000","last_active":"2023-10-03T17:53:38.413061+0000","last_peered":"2023-10-03T17:53:38.413061+0000","last_clean":"2023-10-03T17:53:38.413061+0000","last_became_active":"2023-10-03T17:34:42.276624+0000","last_became_peered":"2023-10-03T17:34:42.276624+0000","last_unstale":"2023-10-03T17:53:38.413061+0000","last_undegraded":"2023-10-03T17:53:38.413061+0000","last_fullsized":"2023-10-03T17:53:38.413061+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'673","last_scrub_stamp":"2023-10-03T17:40:17.406813+0000","last_deep_scrub":"46'673","last_deep_scrub_stamp":"2023-10-03T17:40:17.406813+0000","last_clean_scrub_stamp":"2023-10-03T17:40:17.406813+0000","objects_scrubbed":7,"log_size":2254,"log_dups_size":0,"ondisk_log_size":2254,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T21:38:35.056889+0000","scrub_duration":0.013778076,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":639,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":8311,"num_read_kb":6871,"num_write":363,"num_write_kb":280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":903,"num_omap_keys":30,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"103'3715","reported_seq":9972,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-10-03T17:53:38.427016+0000","last_change":"2023-10-03T17:40:13.376519+0000","last_active":"2023-10-03T17:53:38.427016+0000","last_peered":"2023-10-03T17:53:38.427016+0000","last_clean":"2023-10-03T17:53:38.427016+0000","last_became_active":"2023-10-03T17:34:42.271201+0000","last_became_peered":"2023-10-03T17:34:42.271201+0000","last_unstale":"2023-10-03T17:53:38.427016+0000","last_undegraded":"2023-10-03T17:53:38.427016+0000","last_fullsized":"2023-10-03T17:53:38.427016+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'1129","last_scrub_stamp":"2023-10-03T17:40:13.376332+0000","last_deep_scrub":"46'1129","last_deep_scrub_stamp":"2023-10-03T17:40:13.376332+0000","last_clean_scrub_stamp":"2023-10-03T17:40:13.376332+0000","objects_scrubbed":11,"log_size":3715,"log_dups_size":0,"ondisk_log_size":3715,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T18:59:17.088585+0000","scrub_duration":0.015998675,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":673,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":16443,"num_read_kb":14032,"num_write":592,"num_write_kb":477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2912,"num_omap_keys":93,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"102'66","reported_seq":222,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-10-03T17:53:07.454058+0000","last_change":"2023-10-03T17:40:25.496010+0000","last_active":"2023-10-03T17:53:07.454058+0000","last_peered":"2023-10-03T17:53:07.454058+0000","last_clean":"2023-10-03T17:53:07.454058+0000","last_became_active":"2023-10-03T17:34:42.276941+0000","last_became_peered":"2023-10-03T17:34:42.276941+0000","last_unstale":"2023-10-03T17:53:07.454058+0000","last_undegraded":"2023-10-03T17:53:07.454058+0000","last_fullsized":"2023-10-03T17:53:07.454058+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'14","last_scrub_stamp":"2023-10-03T17:40:25.495827+0000","last_deep_scrub":"46'14","last_deep_scrub_stamp":"2023-10-03T17:40:25.495827+0000","last_clean_scrub_stamp":"2023-10-03T17:40:25.495827+0000","objects_scrubbed":7,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:05:44.939663+0000","scrub_duration":0.0133078,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1057,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":57,"num_read_kb":30,"num_write":59,"num_write_kb":41,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"103'2978","reported_seq":7488,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-10-03T17:53:38.413655+0000","last_change":"2023-10-03T17:40:26.513902+0000","last_active":"2023-10-03T17:53:38.413655+0000","last_peered":"2023-10-03T17:53:38.413655+0000","last_clean":"2023-10-03T17:53:38.413655+0000","last_became_active":"2023-10-03T17:34:42.276459+0000","last_became_peered":"2023-10-03T17:34:42.276459+0000","last_unstale":"2023-10-03T17:53:38.413655+0000","last_undegraded":"2023-10-03T17:53:38.413655+0000","last_fullsized":"2023-10-03T17:53:38.413655+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'964","last_scrub_stamp":"2023-10-03T17:40:26.513729+0000","last_deep_scrub":"46'964","last_deep_scrub_stamp":"2023-10-03T17:40:26.513729+0000","last_clean_scrub_stamp":"2023-10-03T17:40:26.513729+0000","objects_scrubbed":10,"log_size":2978,"log_dups_size":0,"ondisk_log_size":2978,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:41:33.182751+0000","scrub_duration":0.018190280999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":333,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":12080,"num_read_kb":10110,"num_write":452,"num_write_kb":363,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1731,"num_omap_keys":80,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"103'139149","reported_seq":223977,"reported_epoch":103,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T17:53:39.770067+0000","last_change":"2023-10-03T17:53:37.547940+0000","last_active":"2023-10-03T17:53:39.770067+0000","last_peered":"2023-10-03T17:53:39.770067+0000","last_clean":"2023-10-03T17:53:39.770067+0000","last_became_active":"2023-10-03T17:34:48.310701+0000","last_became_peered":"2023-10-03T17:34:48.310701+0000","last_unstale":"2023-10-03T17:53:39.770067+0000","last_undegraded":"2023-10-03T17:53:39.770067+0000","last_fullsized":"2023-10-03T17:53:39.770067+0000","mapping_epoch":16,"log_start":"97'129134","ondisk_log_start":"97'129134","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'80695","last_scrub_stamp":"2023-10-03T17:46:13.982571+0000","last_deep_scrub":"66'80695","last_deep_scrub_stamp":"2023-10-03T17:46:13.982571+0000","last_clean_scrub_stamp":"2023-10-03T17:46:13.982571+0000","objects_scrubbed":3410,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":63,"last_scrub_duration":363,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T01:53:40.177844+0000","scrub_duration":362.26574615599998,"objects_trimmed":471,"snaptrim_duration":30.076963086999999,"stat_sum":{"num_bytes":5724266496,"num_objects":10329,"num_object_clones":9009,"num_object_copies":20658,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10329,"num_whiteouts":0,"num_read":104194,"num_read_kb":395480,"num_write":105408,"num_write_kb":421632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"54","length":"1e"},{"start":"7c","length":"8"},{"start":"8a","length":"2"},{"start":"90","length":"a"},{"start":"a4","length":"a"},{"start":"b8","length":"a"},{"start":"cc","length":"a"},{"start":"e0","length":"a"},{"start":"f4","length":"a"},{"start":"108","length":"a"},{"start":"11c","length":"7"}]},{"pgid":"2.2","version":"103'3086","reported_seq":6674,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-10-03T17:53:37.847489+0000","last_change":"2023-10-03T17:40:08.675119+0000","last_active":"2023-10-03T17:53:37.847489+0000","last_peered":"2023-10-03T17:53:37.847489+0000","last_clean":"2023-10-03T17:53:37.847489+0000","last_became_active":"2023-10-03T17:34:42.270625+0000","last_became_peered":"2023-10-03T17:34:42.270625+0000","last_unstale":"2023-10-03T17:53:37.847489+0000","last_undegraded":"2023-10-03T17:53:37.847489+0000","last_fullsized":"2023-10-03T17:53:37.847489+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'841","last_scrub_stamp":"2023-10-03T17:40:08.674910+0000","last_deep_scrub":"46'841","last_deep_scrub_stamp":"2023-10-03T17:40:08.674910+0000","last_clean_scrub_stamp":"2023-10-03T17:40:08.674910+0000","objects_scrubbed":11,"log_size":3086,"log_dups_size":0,"ondisk_log_size":3086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T21:05:42.790614+0000","scrub_duration":0.039189555000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":857,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":11094,"num_read_kb":9188,"num_write":483,"num_write_kb":383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2564,"num_omap_keys":60,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"103'122284","reported_seq":205196,"reported_epoch":103,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T17:53:39.781818+0000","last_change":"2023-10-03T17:53:39.491295+0000","last_active":"2023-10-03T17:53:39.781818+0000","last_peered":"2023-10-03T17:53:39.781818+0000","last_clean":"2023-10-03T17:53:39.781818+0000","last_became_active":"2023-10-03T17:34:48.310555+0000","last_became_peered":"2023-10-03T17:34:48.310555+0000","last_unstale":"2023-10-03T17:53:39.781818+0000","last_undegraded":"2023-10-03T17:53:39.781818+0000","last_fullsized":"2023-10-03T17:53:39.781818+0000","mapping_epoch":16,"log_start":"96'112241","ondisk_log_start":"96'112241","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"103'121875","last_scrub_stamp":"2023-10-03T17:53:36.632239+0000","last_deep_scrub":"103'121875","last_deep_scrub_stamp":"2023-10-03T17:53:36.632239+0000","last_clean_scrub_stamp":"2023-10-03T17:53:36.632239+0000","objects_scrubbed":9383,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":179,"last_scrub_duration":609,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:22:14.040721+0000","scrub_duration":608.13732607300005,"objects_trimmed":69,"snaptrim_duration":32.023664852000003,"stat_sum":{"num_bytes":5386366976,"num_objects":16784,"num_object_clones":15542,"num_object_copies":33568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16784,"num_whiteouts":0,"num_read":97806,"num_read_kb":370472,"num_write":98320,"num_write_kb":393280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"54","length":"1"}]},{"pgid":"2.1","version":"103'295","reported_seq":2972,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-10-03T17:53:32.730362+0000","last_change":"2023-10-03T17:40:17.641493+0000","last_active":"2023-10-03T17:53:32.730362+0000","last_peered":"2023-10-03T17:53:32.730362+0000","last_clean":"2023-10-03T17:53:32.730362+0000","last_became_active":"2023-10-03T17:34:42.278196+0000","last_became_peered":"2023-10-03T17:34:42.278196+0000","last_unstale":"2023-10-03T17:53:32.730362+0000","last_undegraded":"2023-10-03T17:53:32.730362+0000","last_fullsized":"2023-10-03T17:53:32.730362+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'30","last_scrub_stamp":"2023-10-03T17:40:17.640969+0000","last_deep_scrub":"46'30","last_deep_scrub_stamp":"2023-10-03T17:40:17.640969+0000","last_clean_scrub_stamp":"2023-10-03T17:40:17.640969+0000","objects_scrubbed":6,"log_size":295,"log_dups_size":0,"ondisk_log_size":295,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:04:02.954896+0000","scrub_duration":0.071054565,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":471,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":8272,"num_read_kb":6841,"num_write":337,"num_write_kb":265,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":903,"num_omap_keys":30,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"103'138114","reported_seq":218415,"reported_epoch":103,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T17:53:39.749249+0000","last_change":"2023-10-03T17:53:37.973574+0000","last_active":"2023-10-03T17:53:39.749249+0000","last_peered":"2023-10-03T17:53:39.749249+0000","last_clean":"2023-10-03T17:53:39.749249+0000","last_became_active":"2023-10-03T17:34:48.310912+0000","last_became_peered":"2023-10-03T17:34:48.310912+0000","last_unstale":"2023-10-03T17:53:39.749249+0000","last_undegraded":"2023-10-03T17:53:39.749249+0000","last_fullsized":"2023-10-03T17:53:39.749249+0000","mapping_epoch":16,"log_start":"97'128087","ondisk_log_start":"97'128087","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'58196","last_scrub_stamp":"2023-10-03T17:43:26.193075+0000","last_deep_scrub":"53'58196","last_deep_scrub_stamp":"2023-10-03T17:43:26.193075+0000","last_clean_scrub_stamp":"2023-10-03T17:43:26.193075+0000","objects_scrubbed":2326,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":21,"last_scrub_duration":196,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T04:20:48.866934+0000","scrub_duration":195.44739730800001,"objects_trimmed":577,"snaptrim_duration":30.518614205999999,"stat_sum":{"num_bytes":5515370496,"num_objects":7330,"num_object_clones":6059,"num_object_copies":14660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7330,"num_whiteouts":0,"num_read":100100,"num_read_kb":380776,"num_write":101381,"num_write_kb":405524,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"54","length":"1e"},{"start":"7c","length":"8"},{"start":"8a","length":"2"},{"start":"90","length":"a"},{"start":"a4","length":"a"},{"start":"b8","length":"a"},{"start":"cc","length":"a"},{"start":"e0","length":"a"},{"start":"f4","length":"a"},{"start":"108","length":"a"},{"start":"11c","length":"a"},{"start":"130","length":"a"},{"start":"143","length":"7"},{"start":"150","length":"1"},{"start":"152","length":"1"},{"start":"154","length":"1"},{"start":"158","length":"a"},{"start":"16c","length":"9"}]},{"pgid":"2.0","version":"102'157","reported_seq":1166,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-10-03T17:53:27.612742+0000","last_change":"2023-10-03T17:40:10.708344+0000","last_active":"2023-10-03T17:53:27.612742+0000","last_peered":"2023-10-03T17:53:27.612742+0000","last_clean":"2023-10-03T17:53:27.612742+0000","last_became_active":"2023-10-03T17:34:42.272306+0000","last_became_peered":"2023-10-03T17:34:42.272306+0000","last_unstale":"2023-10-03T17:53:27.612742+0000","last_undegraded":"2023-10-03T17:53:27.612742+0000","last_fullsized":"2023-10-03T17:53:27.612742+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'20","last_scrub_stamp":"2023-10-03T17:40:10.708060+0000","last_deep_scrub":"46'20","last_deep_scrub_stamp":"2023-10-03T17:40:10.708060+0000","last_clean_scrub_stamp":"2023-10-03T17:40:10.708060+0000","objects_scrubbed":7,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:03:09.097252+0000","scrub_duration":0.042279491000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":755,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":2816,"num_read_kb":2315,"num_write":166,"num_write_kb":122,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":301,"num_omap_keys":10,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"103'138303","reported_seq":219961,"reported_epoch":103,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T17:53:39.781821+0000","last_change":"2023-10-03T17:53:35.599606+0000","last_active":"2023-10-03T17:53:39.781821+0000","last_peered":"2023-10-03T17:53:39.781821+0000","last_clean":"2023-10-03T17:53:39.781821+0000","last_became_active":"2023-10-03T17:34:48.310612+0000","last_became_peered":"2023-10-03T17:34:48.310612+0000","last_unstale":"2023-10-03T17:53:39.781821+0000","last_undegraded":"2023-10-03T17:53:39.781821+0000","last_fullsized":"2023-10-03T17:53:39.781821+0000","mapping_epoch":16,"log_start":"97'128289","ondisk_log_start":"97'128289","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'75807","last_scrub_stamp":"2023-10-03T17:45:43.833310+0000","last_deep_scrub":"61'75807","last_deep_scrub_stamp":"2023-10-03T17:45:43.833310+0000","last_clean_scrub_stamp":"2023-10-03T17:45:43.833310+0000","objects_scrubbed":3607,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":42,"last_scrub_duration":315,"scrub_schedule":"queued for deep scrub","scrub_duration":314.26374841099999,"objects_trimmed":482,"snaptrim_duration":28.144488007,"stat_sum":{"num_bytes":5583368211,"num_objects":8754,"num_object_clones":7466,"num_object_copies":17508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8754,"num_whiteouts":0,"num_read":101072,"num_read_kb":384080,"num_write":102546,"num_write_kb":410178,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"54","length":"1e"},{"start":"7c","length":"8"},{"start":"8a","length":"2"},{"start":"90","length":"a"},{"start":"a4","length":"a"},{"start":"b8","length":"a"},{"start":"cc","length":"a"},{"start":"e0","length":"a"},{"start":"f4","length":"a"},{"start":"108","length":"a"},{"start":"11c","length":"a"},{"start":"130","length":"a"},{"start":"143","length":"7"},{"start":"150","length":"1"}]},{"pgid":"2.3","version":"100'55","reported_seq":205,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-10-03T17:53:07.454011+0000","last_change":"2023-10-03T17:40:24.472734+0000","last_active":"2023-10-03T17:53:07.454011+0000","last_peered":"2023-10-03T17:53:07.454011+0000","last_clean":"2023-10-03T17:53:07.454011+0000","last_became_active":"2023-10-03T17:34:42.270194+0000","last_became_peered":"2023-10-03T17:34:42.270194+0000","last_unstale":"2023-10-03T17:53:07.454011+0000","last_undegraded":"2023-10-03T17:53:07.454011+0000","last_fullsized":"2023-10-03T17:53:07.454011+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'8","last_scrub_stamp":"2023-10-03T17:40:24.472573+0000","last_deep_scrub":"46'8","last_deep_scrub_stamp":"2023-10-03T17:40:24.472573+0000","last_clean_scrub_stamp":"2023-10-03T17:40:24.472573+0000","objects_scrubbed":4,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T18:28:04.735033+0000","scrub_duration":0.024990637999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":587,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":45,"num_read_kb":27,"num_write":51,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"46'99","reported_seq":249,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-03T17:53:04.444151+0000","last_change":"2023-10-03T17:40:10.452341+0000","last_active":"2023-10-03T17:53:04.444151+0000","last_peered":"2023-10-03T17:53:04.444151+0000","last_clean":"2023-10-03T17:53:04.444151+0000","last_became_active":"2023-10-03T17:34:39.502393+0000","last_became_peered":"2023-10-03T17:34:39.502393+0000","last_unstale":"2023-10-03T17:53:04.444151+0000","last_undegraded":"2023-10-03T17:53:04.444151+0000","last_fullsized":"2023-10-03T17:53:04.444151+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'99","last_scrub_stamp":"2023-10-03T17:40:10.449099+0000","last_deep_scrub":"46'99","last_deep_scrub_stamp":"2023-10-03T17:40:10.449099+0000","last_clean_scrub_stamp":"2023-10-03T17:40:10.449099+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T20:18:04.958666+0000","scrub_duration":0.043225012,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":22209372179,"num_objects":43197,"num_object_clones":38076,"num_object_copies":86394,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43197,"num_whiteouts":0,"num_read":403172,"num_read_kb":1530808,"num_write":407655,"num_write_kb":1630614,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3334086656,"data_stored":20415057958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40099,"ondisk_log_size":40099,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":5372,"num_objects":143,"num_object_clones":0,"num_object_copies":286,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":143,"num_whiteouts":0,"num_read":59118,"num_read_kb":49414,"num_write":2503,"num_write_kb":1963,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9314,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":983040,"data_stored":10744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7029,"internal_metadata":0},"log_size":12606,"ondisk_log_size":12606,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":7460,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738601,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2943108,"kb_used_data":1625916,"kb_used_omap":2891,"kb_used_meta":1314292,"kb_avail":91428732,"statfs":{"total":96636764160,"available":93623021568,"internally_reserved":0,"allocated":1664937984,"data_stored":10210217041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2960945,"internal_metadata":1345835471},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,4,10,4,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 17:53:08 2023","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.52300000000000002,"15min":0.46400000000000002},"min":{"1min":0.187,"5min":0.158,"15min":0.158},"max":{"1min":1.161,"5min":14.787000000000001,"15min":14.787000000000001},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.41899999999999998,"5min":0.57699999999999996,"15min":0.47299999999999998},"min":{"1min":0.20000000000000001,"5min":0.185,"15min":0.17100000000000001},"max":{"1min":2.9300000000000002,"5min":14.919,"15min":14.919},"last":0.42899999999999999}]},{"osd":1,"last update":"Tue Oct 3 17:53:08 2023","interfaces":[{"interface":"back","average":{"1min":0.36099999999999999,"5min":0.52700000000000002,"15min":0.52500000000000002},"min":{"1min":0.214,"5min":0.214,"15min":0.17599999999999999},"max":{"1min":1.248,"5min":6.7539999999999996,"15min":8.1059999999999999},"last":0.504},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.54100000000000004,"15min":0.48199999999999998},"min":{"1min":0.215,"5min":0.21099999999999999,"15min":0.17699999999999999},"max":{"1min":3.9350000000000001,"5min":5.194,"15min":5.194},"last":0.39400000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738600,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2393312,"kb_used_data":804632,"kb_used_omap":1,"kb_used_meta":1588670,"kb_avail":91978528,"statfs":{"total":96636764160,"available":94186012672,"internally_reserved":0,"allocated":823943168,"data_stored":6015044696,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":1626798534},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1,0,8,12],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 17:53:20 2023","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.36499999999999999,"15min":0.33400000000000002},"min":{"1min":0.187,"5min":0.17599999999999999,"15min":0.161},"max":{"1min":3.823,"5min":3.823,"15min":3.823},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.40699999999999997,"5min":0.35199999999999998,"15min":0.38900000000000001},"min":{"1min":0.217,"5min":0.19500000000000001,"15min":0.16800000000000001},"max":{"1min":1.429,"5min":2.2040000000000002,"15min":5.8959999999999999},"last":0.58599999999999997}]},{"osd":2,"last update":"Tue Oct 3 17:53:20 2023","interfaces":[{"interface":"back","average":{"1min":0.98099999999999998,"5min":0.53800000000000003,"15min":0.44400000000000001},"min":{"1min":0.152,"5min":0.152,"15min":0.152},"max":{"1min":5.7930000000000001,"5min":5.7930000000000001,"15min":5.7930000000000001},"last":0.215},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.40400000000000003,"15min":0.38300000000000001},"min":{"1min":0.24099999999999999,"5min":0.19700000000000001,"15min":0.16400000000000001},"max":{"1min":2.2130000000000001,"5min":2.665,"15min":3.0699999999999998},"last":0.29099999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738600,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2448520,"kb_used_data":827904,"kb_used_omap":1,"kb_used_meta":1620606,"kb_avail":91923320,"statfs":{"total":96636764160,"available":94129479680,"internally_reserved":0,"allocated":847773696,"data_stored":4192687408,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":1659500998},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,0,1,4,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 17:53:05 2023","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.437,"15min":0.42799999999999999},"min":{"1min":0.184,"5min":0.151,"15min":0.151},"max":{"1min":3.1139999999999999,"5min":3.9430000000000001,"15min":5.7649999999999997},"last":0.24099999999999999},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.54100000000000004,"15min":0.48899999999999999},"min":{"1min":0.218,"5min":0.19500000000000001,"15min":0.16900000000000001},"max":{"1min":3.0670000000000002,"5min":3.9169999999999998,"15min":9.4770000000000003},"last":0.438}]},{"osd":2,"last update":"Tue Oct 3 17:53:05 2023","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.49299999999999999,"15min":0.52000000000000002},"min":{"1min":0.218,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":2.3050000000000002,"5min":3.996,"15min":9.4149999999999991},"last":0.27000000000000002},{"interface":"front","average":{"1min":0.36399999999999999,"5min":0.36699999999999999,"15min":0.46899999999999997},"min":{"1min":0.20899999999999999,"5min":0.19600000000000001,"15min":0.189},"max":{"1min":1.179,"5min":1.179,"15min":5.218},"last":0.47199999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3730,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":3559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":5372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1813,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7029,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":847007744,"data_stored":4191088640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":822743040,"data_stored":6011281427,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1664335872,"data_stored":10212687891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-03T17:53:41.303 DEBUG:teuthology.orchestra.run.smithi111:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2023-10-03T17:53:41.918 INFO:teuthology.misc:Shutting down mds daemons... 2023-10-03T17:53:41.919 INFO:teuthology.misc:Shutting down osd daemons... 2023-10-03T17:53:41.919 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2023-10-03T17:53:41.919 INFO:teuthology.orchestra.run:waiting for 300 2023-10-03T17:53:47.331 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~0s 2023-10-03T17:53:48.021 INFO:tasks.ceph.osd.0:Stopped 2023-10-03T17:53:48.022 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2023-10-03T17:53:48.022 INFO:teuthology.orchestra.run:waiting for 300 2023-10-03T17:53:52.634 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~0s 2023-10-03T17:53:52.634 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 has been restored 2023-10-03T17:53:54.123 INFO:tasks.ceph.osd.1:Stopped 2023-10-03T17:53:54.124 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2023-10-03T17:53:54.124 INFO:teuthology.orchestra.run:waiting for 300 2023-10-03T17:53:54.220 INFO:tasks.ceph.osd.2:Stopped 2023-10-03T17:53:54.221 INFO:teuthology.misc:Shutting down mgr daemons... 2023-10-03T17:53:54.221 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2023-10-03T17:53:54.221 INFO:teuthology.orchestra.run:waiting for 300 2023-10-03T17:53:54.245 INFO:tasks.ceph.mgr.x:Stopped 2023-10-03T17:53:54.246 INFO:teuthology.misc:Shutting down mon daemons... 2023-10-03T17:53:54.246 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-10-03T17:53:54.246 INFO:teuthology.orchestra.run:waiting for 300 2023-10-03T17:53:54.299 INFO:tasks.ceph.mon.a:Stopped 2023-10-03T17:53:54.299 INFO:tasks.ceph:Checking cluster log for badness... 2023-10-03T17:53:54.300 DEBUG:teuthology.orchestra.run.smithi111:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 2023-10-03T17:53:54.745 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi111.front.sepia.ceph.com 2023-10-03T17:53:54.745 DEBUG:teuthology.orchestra.run.smithi111:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2023-10-03T17:53:56.943 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi111.front.sepia.ceph.com 2023-10-03T17:53:56.944 DEBUG:teuthology.orchestra.run.smithi111:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2023-10-03T17:53:57.169 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi111.front.sepia.ceph.com 2023-10-03T17:53:57.169 DEBUG:teuthology.orchestra.run.smithi111:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2023-10-03T17:53:57.465 INFO:tasks.ceph:Archiving mon data... 2023-10-03T17:53:57.467 DEBUG:teuthology.misc:Transferring archived files from smithi111:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/rraja-2023-10-03_17:10:36-rbd:cli-wip-ajarr-63009-only-reacquire-kickstart-distro-default-smithi/7409699/data/mon.a.tgz 2023-10-03T17:53:57.467 DEBUG:teuthology.orchestra.run.smithi111:> mktemp 2023-10-03T17:53:57.478 INFO:teuthology.orchestra.run.smithi111.stdout:/tmp/tmp.cgue3QF9pv 2023-10-03T17:53:57.478 DEBUG:teuthology.orchestra.run.smithi111:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.cgue3QF9pv 2023-10-03T17:53:57.620 DEBUG:teuthology.orchestra.run.smithi111:> sudo chmod 0666 /tmp/tmp.cgue3QF9pv 2023-10-03T17:53:57.635 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 has been restored 2023-10-03T17:53:57.700 DEBUG:teuthology.orchestra.remote:smithi111:/tmp/tmp.cgue3QF9pv is 332KB 2023-10-03T17:53:57.762 DEBUG:teuthology.orchestra.run.smithi111:> rm -fr /tmp/tmp.cgue3QF9pv 2023-10-03T17:53:57.778 INFO:tasks.ceph:Cleaning ceph cluster... 2023-10-03T17:53:57.779 DEBUG:teuthology.orchestra.run.smithi111:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-10-03T17:53:57.863 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2023-10-03T17:53:57.863 DEBUG:teuthology.orchestra.run.smithi111:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-10-03T17:53:57.923 INFO:teuthology.orchestra.run.smithi111.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-10-03T17:53:57.927 INFO:tasks.ceph:Archiving crash dumps... 2023-10-03T17:53:57.929 DEBUG:teuthology.misc:Transferring archived files from smithi111:/var/lib/ceph/crash to /home/teuthworker/archive/rraja-2023-10-03_17:10:36-rbd:cli-wip-ajarr-63009-only-reacquire-kickstart-distro-default-smithi/7409699/remote/smithi111/crash 2023-10-03T17:53:57.929 DEBUG:teuthology.orchestra.run.smithi111:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-10-03T17:53:57.978 INFO:tasks.ceph:Compressing logs... 2023-10-03T17:53:57.978 DEBUG:teuthology.orchestra.run.smithi111:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-10-03T17:53:58.037 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41645.log 2023-10-03T17:53:58.037 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38645.log 2023-10-03T17:53:58.040 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.41645.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37422.log 2023-10-03T17:53:58.040 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41645.log.gz 2023-10-03T17:53:58.040 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41323.log 2023-10-03T17:53:58.040 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.38645.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38645.log.gz 2023-10-03T17:53:58.040 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.37422.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34385.log 2023-10-03T17:53:58.040 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37422.log.gz 2023-10-03T17:53:58.041 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.41323.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39111.log 2023-10-03T17:53:58.041 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41323.log.gz 2023-10-03T17:53:58.041 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39165.log 2023-10-03T17:53:58.041 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.34385.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34385.log.gz 2023-10-03T17:53:58.041 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.39111.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40764.log 2023-10-03T17:53:58.041 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39111.log.gz 2023-10-03T17:53:58.041 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.39165.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39165.log.gz 2023-10-03T17:53:58.041 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42446.log 2023-10-03T17:53:58.050 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.40764.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40764.log.gz 2023-10-03T17:53:58.050 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38247.log 2023-10-03T17:53:58.050 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.42446.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37648.log 2023-10-03T17:53:58.050 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42446.log.gz 2023-10-03T17:53:58.050 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.38247.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38318.log 2023-10-03T17:53:58.050 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38247.log.gz 2023-10-03T17:53:58.050 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.37648.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34409.log 2023-10-03T17:53:58.051 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37648.log.gz 2023-10-03T17:53:58.051 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.38318.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40521.log 2023-10-03T17:53:58.051 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38318.log.gz 2023-10-03T17:53:58.051 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.34409.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39273.log 2023-10-03T17:53:58.051 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34409.log.gz 2023-10-03T17:53:58.051 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.40521.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41807.log 2023-10-03T17:53:58.051 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40521.log.gz 2023-10-03T17:53:58.052 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39739.log 2023-10-03T17:53:58.052 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.39273.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39273.log.gz 2023-10-03T17:53:58.052 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.41807.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39991.log 2023-10-03T17:53:58.052 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41807.log.gz 2023-10-03T17:53:58.052 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2023-10-03T17:53:58.052 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.39739.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39739.log.gz 2023-10-03T17:53:58.052 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.39991.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42180.log 2023-10-03T17:53:58.052 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39991.log.gz 2023-10-03T17:53:58.053 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37725.log 2023-10-03T17:53:58.053 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.42180.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42233.log 2023-10-03T17:53:58.053 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42180.log.gz 2023-10-03T17:53:58.053 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.37725.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38932.log 2023-10-03T17:53:58.053 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37725.log.gz 2023-10-03T17:53:58.053 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.42233.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37421.log 2023-10-03T17:53:58.053 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42233.log.gz 2023-10-03T17:53:58.053 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.38932.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38770.log 2023-10-03T17:53:58.054 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38932.log.gz 2023-10-03T17:53:58.054 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.37421.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38699.log 2023-10-03T17:53:58.054 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37421.log.gz 2023-10-03T17:53:58.054 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.38770.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38554.log 0.0% 2023-10-03T17:53:58.054 INFO:teuthology.orchestra.run.smithi111.stderr: -- replaced with /var/log/ceph/ceph-client.admin.38770.log.gz 2023-10-03T17:53:58.054 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.38699.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41859.log 2023-10-03T17:53:58.054 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38699.log.gz 2023-10-03T17:53:58.055 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.38554.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40302.log 2023-10-03T17:53:58.055 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38554.log.gz 2023-10-03T17:53:58.055 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.41859.log: gzip -5 0.0% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.41859.log.gz 2023-10-03T17:53:58.055 INFO:teuthology.orchestra.run.smithi111.stderr: /var/log/ceph/ceph-client.admin.42500.log 2023-10-03T17:53:58.055 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.40302.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39875.log 2023-10-03T17:53:58.055 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40302.log.gz 2023-10-03T17:53:58.055 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.42500.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41460.log 2023-10-03T17:53:58.055 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42500.log.gz 2023-10-03T17:53:58.056 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.39875.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39185.log 2023-10-03T17:53:58.056 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39875.log.gz 2023-10-03T17:53:58.056 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.41460.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.30709.log 2023-10-03T17:53:58.056 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41460.log.gz 2023-10-03T17:53:58.056 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.39185.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41115.log 2023-10-03T17:53:58.056 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39185.log.gz 2023-10-03T17:53:58.056 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph.tmp-client.admin.30709.log: gzip 0.0% -5 -- replaced with /var/log/ceph/ceph.tmp-client.admin.30709.log.gz --verbose 2023-10-03T17:53:58.056 INFO:teuthology.orchestra.run.smithi111.stderr: -- /var/log/ceph/ceph-client.admin.37700.log 2023-10-03T17:53:58.104 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.41115.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41430.log 2023-10-03T17:53:58.104 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41115.log.gz 2023-10-03T17:53:58.104 INFO:teuthology.orchestra.run.smithi111.stderr:gzip/var/log/ceph/ceph-client.admin.37700.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.42419.log 2023-10-03T17:53:58.104 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37700.log.gz 2023-10-03T17:53:58.104 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.41430.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38446.log 2023-10-03T17:53:58.104 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41430.log.gz 2023-10-03T17:53:58.104 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.42419.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42047.log 2023-10-03T17:53:58.105 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42419.log.gz 2023-10-03T17:53:58.105 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.38446.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2023-10-03T17:53:58.105 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38446.log.gz 2023-10-03T17:53:58.105 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.42047.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39023.log 2023-10-03T17:53:58.105 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42047.log.gz 2023-10-03T17:53:58.105 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38986.log 2023-10-03T17:53:58.105 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38338.log 2023-10-03T17:53:58.106 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.39023.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39023.log.gz 2023-10-03T17:53:58.106 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.38986.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38986.log.gz 2023-10-03T17:53:58.106 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42367.log 2023-10-03T17:53:58.106 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.38338.log: /var/log/ceph/ceph-client.admin.34510.log 2023-10-03T17:53:58.106 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38338.log.gz 2023-10-03T17:53:58.106 INFO:teuthology.orchestra.run.smithi111.stderr:gzip/var/log/ceph/ceph-client.admin.42367.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.41697.log 2023-10-03T17:53:58.106 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42367.log.gz 2023-10-03T17:53:58.106 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.34510.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41833.log 2023-10-03T17:53:58.107 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34510.log.gz 2023-10-03T17:53:58.107 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.41697.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38824.log 2023-10-03T17:53:58.107 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41697.log.gz 2023-10-03T17:53:58.107 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.41833.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37577.log 2023-10-03T17:53:58.107 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41833.log.gz 2023-10-03T17:53:58.107 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.38824.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38787.log 2023-10-03T17:53:58.107 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38824.log.gz 2023-10-03T17:53:58.107 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.37577.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39131.log 2023-10-03T17:53:58.108 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37577.log.gz 2023-10-03T17:53:58.108 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.38787.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2023-10-03T17:53:58.108 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38787.log.gz 2023-10-03T17:53:58.108 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.39131.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42339.log 2023-10-03T17:53:58.108 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39131.log.gz 2023-10-03T17:53:58.108 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41781.log 2023-10-03T17:53:58.108 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.42339.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37263.log 2023-10-03T17:53:58.108 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42339.log.gz 2023-10-03T17:53:58.109 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.41781.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42101.log 2023-10-03T17:53:58.109 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41781.log.gz 2023-10-03T17:53:58.109 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.37263.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39293.log 2023-10-03T17:53:58.109 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37263.log.gz 2023-10-03T17:53:58.109 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.42101.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41912.log 2023-10-03T17:53:58.109 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42101.log.gz 2023-10-03T17:53:58.109 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.39293.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40882.log 2023-10-03T17:53:58.109 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39293.log.gz 2023-10-03T17:53:58.110 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.41912.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39239.log 2023-10-03T17:53:58.110 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41912.log.gz 2023-10-03T17:53:58.110 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.40882.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41167.log 2023-10-03T17:53:58.110 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40882.log.gz 2023-10-03T17:53:58.110 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.39239.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37578.log 2023-10-03T17:53:58.110 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39239.log.gz 2023-10-03T17:53:58.110 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.41167.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40469.log 2023-10-03T17:53:58.110 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41167.log.gz 2023-10-03T17:53:58.111 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.37578.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38298.log 2023-10-03T17:53:58.111 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37578.log.gz 2023-10-03T17:53:58.111 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.40469.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39413.log 2023-10-03T17:53:58.111 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40469.log.gz 2023-10-03T17:53:58.111 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.38298.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41349.log 2023-10-03T17:53:58.111 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38298.log.gz 2023-10-03T17:53:58.111 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.39413.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40548.log 2023-10-03T17:53:58.111 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39413.log.gz 2023-10-03T17:53:58.112 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.41349.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38591.log 2023-10-03T17:53:58.112 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41349.log.gz 2023-10-03T17:53:58.112 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.40548.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39949.log 2023-10-03T17:53:58.112 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40548.log.gz 2023-10-03T17:53:58.112 INFO:teuthology.orchestra.run.smithi111.stderr:gzip/var/log/ceph/ceph-client.admin.38591.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.42287.log 2023-10-03T17:53:58.112 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38591.log.gz 2023-10-03T17:53:58.112 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.39949.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41063.log 2023-10-03T17:53:58.112 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39949.log.gz 2023-10-03T17:53:58.113 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.42287.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41538.log 2023-10-03T17:53:58.113 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42287.log.gz 2023-10-03T17:53:58.113 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.41063.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37336.log 2023-10-03T17:53:58.113 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41063.log.gz 2023-10-03T17:53:58.113 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.41538.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34361.log 2023-10-03T17:53:58.113 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41538.log.gz 2023-10-03T17:53:58.113 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.37336.log: gzip -5 0.0% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.37336.log.gz 2023-10-03T17:53:58.113 INFO:teuthology.orchestra.run.smithi111.stderr: /var/log/ceph/ceph-client.admin.37674.log 2023-10-03T17:53:58.114 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.34361.log: -- /var/log/ceph/ceph-client.admin.38625.log 2023-10-03T17:53:58.114 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34361.log.gz 2023-10-03T17:53:58.114 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.37674.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42393.log 2023-10-03T17:53:58.114 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37674.log.gz 2023-10-03T17:53:58.114 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.38625.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38571.log 2023-10-03T17:53:58.114 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38625.log.gz 2023-10-03T17:53:58.114 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.42393.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37075.log 2023-10-03T17:53:58.115 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42393.log.gz 2023-10-03T17:53:58.115 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.38571.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38571.log.gzgzip 2023-10-03T17:53:58.115 INFO:teuthology.orchestra.run.smithi111.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.38662.log 2023-10-03T17:53:58.115 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.37075.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37419.log 2023-10-03T17:53:58.115 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37075.log.gz 2023-10-03T17:53:58.115 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.38662.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40395.log 2023-10-03T17:53:58.115 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38662.log.gz 2023-10-03T17:53:58.115 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.37419.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42261.log 2023-10-03T17:53:58.116 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37419.log.gz 2023-10-03T17:53:58.116 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.40395.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38915.log 2023-10-03T17:53:58.116 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40395.log.gz 2023-10-03T17:53:58.116 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.42261.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41089.log 2023-10-03T17:53:58.116 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42261.log.gz 2023-10-03T17:53:58.116 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.38915.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37492.log 2023-10-03T17:53:58.116 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38915.log.gz 2023-10-03T17:53:58.117 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.41089.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40913.log 2023-10-03T17:53:58.117 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41089.log.gz 2023-10-03T17:53:58.117 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.37492.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37499.log 2023-10-03T17:53:58.117 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37492.log.gz 2023-10-03T17:53:58.117 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.40913.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41011.log 2023-10-03T17:53:58.117 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40913.log.gz 2023-10-03T17:53:58.117 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.37499.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37499.log.gz 2023-10-03T17:53:58.117 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34267.log 2023-10-03T17:53:58.118 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.41011.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41219.log 2023-10-03T17:53:58.118 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41011.log.gz 2023-10-03T17:53:58.118 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.34267.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.39525.log 2023-10-03T17:53:58.118 INFO:teuthology.orchestra.run.smithi111.stderr: -- replaced with /var/log/ceph/ceph-client.admin.34267.log.gz 2023-10-03T17:53:58.118 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42074.log 2023-10-03T17:53:58.118 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.41219.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41219.log.gz 2023-10-03T17:53:58.118 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.39525.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40668.log 2023-10-03T17:53:58.119 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39525.log.gz 2023-10-03T17:53:58.119 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.42074.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38807.log 2023-10-03T17:53:58.119 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42074.log.gz 2023-10-03T17:53:58.119 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.40668.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40223.log 2023-10-03T17:53:58.119 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40668.log.gz 2023-10-03T17:53:58.119 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.38807.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38355.log 2023-10-03T17:53:58.119 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38807.log.gz 2023-10-03T17:53:58.119 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.40223.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41402.log 2023-10-03T17:53:58.120 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40223.log.gz 2023-10-03T17:53:58.120 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.38355.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34444.log 2023-10-03T17:53:58.120 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38355.log.gz 2023-10-03T17:53:58.120 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37751.log 2023-10-03T17:53:58.120 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.41402.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41402.log.gz 2023-10-03T17:53:58.120 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.34444.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41486.log 2023-10-03T17:53:58.120 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34444.log.gz 2023-10-03T17:53:58.120 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.37751.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38861.log 2023-10-03T17:53:58.121 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37751.log.gz 2023-10-03T17:53:58.121 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2023-10-03T17:53:58.121 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.41486.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41486.log.gz 2023-10-03T17:53:58.121 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.38861.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41564.log 2023-10-03T17:53:58.121 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38861.log.gz 2023-10-03T17:53:58.121 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39310.log 2023-10-03T17:53:58.121 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.41564.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39256.log 2023-10-03T17:53:58.121 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41564.log.gz 2023-10-03T17:53:58.122 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.39310.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39219.log 2023-10-03T17:53:58.122 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39310.log.gz 2023-10-03T17:53:58.122 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.39256.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40821.log 2023-10-03T17:53:58.122 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39256.log.gz 2023-10-03T17:53:58.122 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.39219.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41671.log 2023-10-03T17:53:58.122 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39219.log.gz 2023-10-03T17:53:58.122 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.40821.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40425.log 2023-10-03T17:53:58.123 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40821.log.gz 2023-10-03T17:53:58.123 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.41671.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42604.log 2023-10-03T17:53:58.123 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41671.log.gz 2023-10-03T17:53:58.123 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.40425.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39445.log 2023-10-03T17:53:58.123 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40425.log.gz 2023-10-03T17:53:58.123 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.42604.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39781.log 2023-10-03T17:53:58.123 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42604.log.gz 2023-10-03T17:53:58.123 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.39445.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38375.log 2023-10-03T17:53:58.124 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39445.log.gz 2023-10-03T17:53:58.124 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.39781.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38733.log 2023-10-03T17:53:58.124 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39781.log.gz 2023-10-03T17:53:58.124 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.38375.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38716.log 2023-10-03T17:53:58.124 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38375.log.gz 2023-10-03T17:53:58.124 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.38733.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38753.log 2023-10-03T17:53:58.124 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38733.log.gz 2023-10-03T17:53:58.124 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.38716.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38841.log 2023-10-03T17:53:58.125 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38716.log.gz 2023-10-03T17:53:58.125 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.38753.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2023-10-03T17:53:58.125 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38753.log.gz 2023-10-03T17:53:58.125 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.38841.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37570.log 2023-10-03T17:53:58.125 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38841.log.gz 2023-10-03T17:53:58.125 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41991.log 2023-10-03T17:53:58.125 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.37570.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2023-10-03T17:53:58.125 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37570.log.gz 2023-10-03T17:53:58.126 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38463.log 2023-10-03T17:53:58.126 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.41991.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41991.log.gz 2023-10-03T17:53:58.126 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38429.log 2023-10-03T17:53:58.126 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.38463.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42127.log 2023-10-03T17:53:58.126 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38463.log.gz 2023-10-03T17:53:58.126 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.38429.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38429.log.gz 2023-10-03T17:53:58.126 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38281.log 2023-10-03T17:53:58.126 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.42127.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42127.log.gz 2023-10-03T17:53:58.127 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38392.log 2023-10-03T17:53:58.127 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.38281.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41752.log 2023-10-03T17:53:58.127 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38281.log.gz 2023-10-03T17:53:58.127 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.38392.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38517.log 2023-10-03T17:53:58.127 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38392.log.gz 2023-10-03T17:53:58.127 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.41752.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39327.log 2023-10-03T17:53:58.127 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41752.log.gz 2023-10-03T17:53:58.128 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.38517.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38949.log 2023-10-03T17:53:58.128 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38517.log.gz 2023-10-03T17:53:58.128 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.39327.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41964.log 2023-10-03T17:53:58.128 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39327.log.gz 2023-10-03T17:53:58.128 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.38949.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38949.log.gz 2023-10-03T17:53:58.128 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42527.log 2023-10-03T17:53:58.128 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.41964.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41725.log 2023-10-03T17:53:58.128 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41964.log.gz 2023-10-03T17:53:58.128 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.42527.log: gzip 0.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.42527.log.gz --verbose 2023-10-03T17:53:58.129 INFO:teuthology.orchestra.run.smithi111.stderr: -- /var/log/ceph/ceph-client.admin.40608.log 2023-10-03T17:53:58.129 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.41725.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39077.log 2023-10-03T17:53:58.129 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41725.log.gz 2023-10-03T17:53:58.129 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.40608.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41512.log 2023-10-03T17:53:58.129 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40608.log.gz 2023-10-03T17:53:58.129 INFO:teuthology.orchestra.run.smithi111.stderr: 89.1% -- replaced with /var/log/ceph/ceph.log.gz 2023-10-03T17:53:58.129 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.39077.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42313.log 2023-10-03T17:53:58.129 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39077.log.gz 2023-10-03T17:53:58.130 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.41512.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39040.log 2023-10-03T17:53:58.130 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41512.log.gz 2023-10-03T17:53:58.130 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.42313.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40362.log 2023-10-03T17:53:58.130 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42313.log.gz 2023-10-03T17:53:58.130 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.39040.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38500.log 2023-10-03T17:53:58.130 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39040.log.gz 2023-10-03T17:53:58.130 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.40362.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38895.log 2023-10-03T17:53:58.130 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40362.log.gz 2023-10-03T17:53:58.131 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.38500.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37235.log 2023-10-03T17:53:58.131 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38500.log.gz 2023-10-03T17:53:58.131 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.38895.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41141.log 2023-10-03T17:53:58.131 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38895.log.gz 2023-10-03T17:53:58.131 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.37235.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37235.log.gz 2023-10-03T17:53:58.131 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37500.log 2023-10-03T17:53:58.131 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.41141.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38878.log 2023-10-03T17:53:58.131 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41141.log.gz 2023-10-03T17:53:58.131 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39148.log 2023-10-03T17:53:58.132 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.37500.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37500.log.gz 2023-10-03T17:53:58.132 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.38878.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38878.log.gz 2023-10-03T17:53:58.132 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37362.log 2023-10-03T17:53:58.132 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.39148.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39365.log 2023-10-03T17:53:58.132 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39148.log.gz 2023-10-03T17:53:58.132 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.37362.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39202.log 2023-10-03T17:53:58.132 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37362.log.gz 2023-10-03T17:53:58.133 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.39365.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41297.log 2023-10-03T17:53:58.133 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39365.log.gz 2023-10-03T17:53:58.133 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.39202.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40497.log 0.0% 2023-10-03T17:53:58.133 INFO:teuthology.orchestra.run.smithi111.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39202.log.gz 2023-10-03T17:53:58.133 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.41297.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40851.log 2023-10-03T17:53:58.133 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41297.log.gz 2023-10-03T17:53:58.133 INFO:teuthology.orchestra.run.smithi111.stderr:gzip/var/log/ceph/ceph-client.admin.40497.log: -5 --verbose -- /var/log/ceph/ceph.audit.log 2023-10-03T17:53:58.133 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40497.log.gz 2023-10-03T17:53:58.134 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.40851.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40740.log 2023-10-03T17:53:58.134 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40851.log.gz 2023-10-03T17:53:58.134 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42154.log 2023-10-03T17:53:58.134 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.40740.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40985.log 2023-10-03T17:53:58.134 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40740.log.gz 2023-10-03T17:53:58.134 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.42154.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41375.log 2023-10-03T17:53:58.134 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42154.log.gz 2023-10-03T17:53:58.135 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.40985.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39003.log 2023-10-03T17:53:58.135 INFO:teuthology.orchestra.run.smithi111.stderr: 26.9% -- replaced with /var/log/ceph/ceph-client.admin.40985.log.gz 2023-10-03T17:53:58.135 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.41375.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39569.log 2023-10-03T17:53:58.135 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41375.log.gz 2023-10-03T17:53:58.135 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.39003.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40638.log 2023-10-03T17:53:58.135 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39003.log.gz 2023-10-03T17:53:58.135 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.39569.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38264.log 2023-10-03T17:53:58.135 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39569.log.gz 2023-10-03T17:53:58.136 INFO:teuthology.orchestra.run.smithi111.stderr:gzip/var/log/ceph/ceph-client.admin.40638.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.42580.log 2023-10-03T17:53:58.136 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40638.log.gz 2023-10-03T17:53:58.136 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.38264.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42474.log 2023-10-03T17:53:58.136 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38264.log.gz 2023-10-03T17:53:58.136 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.42580.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41037.log 2023-10-03T17:53:58.136 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42580.log.gz 2023-10-03T17:53:58.136 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.42474.log: /var/log/ceph/ceph-client.admin.42206.log 2023-10-03T17:53:58.136 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42474.log.gz 2023-10-03T17:53:58.137 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.41037.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41037.log.gz 2023-10-03T17:53:58.137 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39094.log 2023-10-03T17:53:58.137 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.42206.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42206.log.gz 2023-10-03T17:53:58.137 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39655.log 2023-10-03T17:53:58.137 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.39094.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38483.log 2023-10-03T17:53:58.137 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39094.log.gz 2023-10-03T17:53:58.137 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.39655.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37388.log 2023-10-03T17:53:58.137 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39655.log.gz 2023-10-03T17:53:58.138 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.38483.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40085.log 2023-10-03T17:53:58.138 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38483.log.gz 2023-10-03T17:53:58.138 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41938.log 2023-10-03T17:53:58.138 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.37388.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37388.log.gz 2023-10-03T17:53:58.138 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.40085.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39477.log 2023-10-03T17:53:58.138 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40085.log.gz 2023-10-03T17:53:58.138 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.41938.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40251.log 2023-10-03T17:53:58.138 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41938.log.gz 2023-10-03T17:53:58.139 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.39477.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38537.log 2023-10-03T17:53:58.139 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39477.log.gz 2023-10-03T17:53:58.139 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.40251.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41193.log 2023-10-03T17:53:58.139 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40251.log.gz 2023-10-03T17:53:58.139 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.38537.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41591.log 2023-10-03T17:53:58.139 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38537.log.gz 2023-10-03T17:53:58.139 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.41193.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40791.log 2023-10-03T17:53:58.139 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41193.log.gz 2023-10-03T17:53:58.140 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.41591.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38969.log 2023-10-03T17:53:58.140 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41591.log.gz 2023-10-03T17:53:58.140 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.40791.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37310.log 2023-10-03T17:53:58.140 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40791.log.gz 2023-10-03T17:53:58.140 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.38969.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39823.log 2023-10-03T17:53:58.140 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38969.log.gz 2023-10-03T17:53:58.140 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.37310.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40578.log 2023-10-03T17:53:58.140 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37310.log.gz 2023-10-03T17:53:58.140 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.39823.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39823.log.gzgzip 2023-10-03T17:53:58.141 INFO:teuthology.orchestra.run.smithi111.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.38608.log 2023-10-03T17:53:58.141 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.40578.log: /var/log/ceph/ceph-client.admin.40332.log 2023-10-03T17:53:58.141 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40578.log.gz 2023-10-03T17:53:58.141 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.38608.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38409.log 2023-10-03T17:53:58.141 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38608.log.gz 2023-10-03T17:53:58.141 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.40332.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.40332.log.gz 2023-10-03T17:53:58.141 INFO:teuthology.orchestra.run.smithi111.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.40041.log 2023-10-03T17:53:58.141 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.38409.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40953.log 2023-10-03T17:53:58.142 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38409.log.gz 2023-10-03T17:53:58.142 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42020.log 2023-10-03T17:53:58.142 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.40041.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40041.log.gz 2023-10-03T17:53:58.142 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.40953.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40712.log 2023-10-03T17:53:58.142 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40953.log.gz 2023-10-03T17:53:58.142 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.42020.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39347.log 2023-10-03T17:53:58.142 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42020.log.gz 2023-10-03T17:53:58.143 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.40712.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39612.log 2023-10-03T17:53:58.143 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40712.log.gz 2023-10-03T17:53:58.143 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.39347.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40167.log 2023-10-03T17:53:58.143 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39347.log.gz 2023-10-03T17:53:58.143 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.39612.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39057.log 2023-10-03T17:53:58.143 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39612.log.gz 2023-10-03T17:53:58.143 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.40167.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41271.log 2023-10-03T17:53:58.143 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40167.log.gz 2023-10-03T17:53:58.144 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.39057.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41885.log 2023-10-03T17:53:58.144 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39057.log.gz 2023-10-03T17:53:58.144 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.41271.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40125.log 2023-10-03T17:53:58.144 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41271.log.gz 2023-10-03T17:53:58.144 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.41885.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42554.log 2023-10-03T17:53:58.144 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41885.log.gz 2023-10-03T17:53:58.144 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.40125.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39916.log 2023-10-03T17:53:58.144 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40125.log.gz 2023-10-03T17:53:58.145 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.42554.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39697.log 2023-10-03T17:53:58.145 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42554.log.gz 2023-10-03T17:53:58.145 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.39916.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41618.log 2023-10-03T17:53:58.145 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39916.log.gz 2023-10-03T17:53:58.145 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.39697.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40275.log 2023-10-03T17:53:58.145 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39697.log.gz 2023-10-03T17:53:58.145 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.41618.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41245.log 2023-10-03T17:53:58.145 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41618.log.gz 2023-10-03T17:53:58.146 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.40275.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38679.log 2023-10-03T17:53:58.146 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40275.log.gz 2023-10-03T17:53:58.146 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.41245.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41245.log.gz 2023-10-03T17:53:58.146 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.38679.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38679.log.gz 2023-10-03T17:53:58.706 INFO:teuthology.orchestra.run.smithi111.stderr: 89.1% -- replaced with /var/log/ceph/ceph.audit.log.gz 2023-10-03T17:53:58.887 INFO:teuthology.orchestra.run.smithi111.stderr: 91.4% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2023-10-03T17:53:59.237 INFO:teuthology.orchestra.run.smithi111.stderr: 90.3% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2023-10-03T17:55:18.163 INFO:teuthology.orchestra.run.smithi111.stderr: 94.2% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2023-10-03T17:56:13.810 INFO:teuthology.orchestra.run.smithi111.stderr: 94.1% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2023-10-03T17:56:18.206 INFO:teuthology.orchestra.run.smithi111.stderr: 94.2% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2023-10-03T17:56:18.207 INFO:teuthology.orchestra.run.smithi111.stderr: 2023-10-03T17:56:18.208 INFO:teuthology.orchestra.run.smithi111.stderr:real 2m20.182s 2023-10-03T17:56:18.208 INFO:teuthology.orchestra.run.smithi111.stderr:user 4m24.138s 2023-10-03T17:56:18.208 INFO:teuthology.orchestra.run.smithi111.stderr:sys 0m11.240s 2023-10-03T17:56:18.208 INFO:tasks.ceph:Archiving logs... 2023-10-03T17:56:18.210 DEBUG:teuthology.misc:Transferring archived files from smithi111:/var/log/ceph to /home/teuthworker/archive/rraja-2023-10-03_17:10:36-rbd:cli-wip-ajarr-63009-only-reacquire-kickstart-distro-default-smithi/7409699/remote/smithi111/log 2023-10-03T17:56:18.211 DEBUG:teuthology.orchestra.run.smithi111:> sudo tar c -f - -C /var/log/ceph -- . 2023-10-03T17:56:41.276 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-10-03T17:56:41.286 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-10-03T17:56:41.286 DEBUG:teuthology.orchestra.run.smithi111:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-10-03T17:56:41.307 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-10-03T17:56:41.316 INFO:teuthology.task.clock:Checking final clock skew... 2023-10-03T17:56:41.316 DEBUG:teuthology.orchestra.run.smithi111:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-03T17:56:41.468 INFO:teuthology.orchestra.run.smithi111.stdout: remote refid st t when poll reach delay offset jitter 2023-10-03T17:56:41.468 INFO:teuthology.orchestra.run.smithi111.stdout:============================================================================== 2023-10-03T17:56:41.469 INFO:teuthology.orchestra.run.smithi111.stdout:*hv01.front.sepi 44.4.53.2 2 u 9 64 377 0.093 -5.681 0.685 2023-10-03T17:56:41.469 INFO:teuthology.orchestra.run.smithi111.stdout:+hv02.front.sepi 63.231.80.2 3 u 2 64 377 0.067 -4.586 0.528 2023-10-03T17:56:41.469 INFO:teuthology.orchestra.run.smithi111.stdout:+hv03.front.sepi 74.6.168.72 3 u 15 64 377 0.081 -2.175 0.593 2023-10-03T17:56:41.469 INFO:teuthology.orchestra.run.smithi111.stdout: hv04.front.sepi .INIT. 16 u - 512 0 0.000 +0.000 0.000 2023-10-03T17:56:41.470 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-10-03T17:56:41.479 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-10-03T17:56:41.479 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-10-03T17:56:41.487 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-10-03T17:56:41.495 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-10-03T17:56:41.504 INFO:teuthology.task.internal:Duration was 2041.274386 seconds 2023-10-03T17:56:41.504 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-10-03T17:56:41.512 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-10-03T17:56:41.512 DEBUG:teuthology.orchestra.run.smithi111:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-10-03T17:56:41.741 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-10-03T17:56:41.742 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi111.front.sepia.ceph.com 2023-10-03T17:56:41.742 DEBUG:teuthology.orchestra.run.smithi111:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-10-03T17:56:41.752 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-10-03T17:56:41.752 DEBUG:teuthology.orchestra.run.smithi111:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-10-03T17:56:41.841 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-10-03T17:56:41.841 DEBUG:teuthology.orchestra.run.smithi111:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-10-03T17:56:42.103 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-10-03T17:56:42.114 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-10-03T17:56:42.114 DEBUG:teuthology.orchestra.run.smithi111:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-10-03T17:56:42.149 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-10-03T17:56:42.159 DEBUG:teuthology.orchestra.run.smithi111:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-10-03T17:56:42.235 INFO:teuthology.orchestra.run.smithi111.stdout:kernel.core_pattern = core 2023-10-03T17:56:42.269 DEBUG:teuthology.orchestra.run.smithi111:> test -e /home/ubuntu/cephtest/archive/coredump 2023-10-03T17:56:42.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-03T17:56:42.318 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-10-03T17:56:42.328 INFO:teuthology.task.internal:Transferring archived files... 2023-10-03T17:56:42.329 DEBUG:teuthology.misc:Transferring archived files from smithi111:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/rraja-2023-10-03_17:10:36-rbd:cli-wip-ajarr-63009-only-reacquire-kickstart-distro-default-smithi/7409699/remote/smithi111 2023-10-03T17:56:42.329 DEBUG:teuthology.orchestra.run.smithi111:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-10-03T17:56:42.385 INFO:teuthology.task.internal:Removing archive directory... 2023-10-03T17:56:42.385 DEBUG:teuthology.orchestra.run.smithi111:> rm -rf -- /home/ubuntu/cephtest/archive 2023-10-03T17:56:42.435 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-10-03T17:56:42.446 INFO:teuthology.task.internal:Not uploading archives. 2023-10-03T17:56:42.446 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-10-03T17:56:42.455 INFO:teuthology.task.internal:Tidying up after the test... 2023-10-03T17:56:42.455 DEBUG:teuthology.orchestra.run.smithi111:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-10-03T17:56:42.483 INFO:teuthology.orchestra.run.smithi111.stdout: 655472 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Oct 3 17:56 /home/ubuntu/cephtest 2023-10-03T17:56:42.483 INFO:teuthology.orchestra.run.smithi111.stdout: 673423 4 drwxrwxr-x 3 ubuntu ubuntu 4096 Oct 3 17:35 /home/ubuntu/cephtest/mnt.0 2023-10-03T17:56:42.483 INFO:teuthology.orchestra.run.smithi111.stdout: 673848 4 drwxrwxr-x 3 ubuntu ubuntu 4096 Oct 3 17:36 /home/ubuntu/cephtest/mnt.0/client.0 2023-10-03T17:56:42.483 INFO:teuthology.orchestra.run.smithi111.stdout: 687731 4 drwxrwxr-x 2 ubuntu ubuntu 4096 Oct 3 17:36 /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-10-03T17:56:42.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-03T17:56:42.484 INFO:teuthology.orchestra.run.smithi111.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2023-10-03T17:56:42.485 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/task/internal/__init__.py", line 53, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi111 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2023-10-03T17:56:42.492 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-10-03T17:56:42.502 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-10-03T17:56:42.512 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: CommandFailedError: Command failed on smithi111 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2023-10-03T17:56:42.512 INFO:teuthology.nuke:Checking targets against current locks 2023-10-03T17:56:42.545 DEBUG:teuthology.nuke:shortname: smithi111 2023-10-03T17:56:42.546 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-03T17:56:42.568 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi111.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rraja-2023-10-03_17:10:36-rbd:cli-wip-ajarr-63009-only-reacquire-kickstart-distro-default-smithi/7409699', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-03 17:11:37.057710', 'locked_by': 'scheduled_rraja@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEnvb/71JGXyR+c0QUpVhUkGSLg+nELPCl6VdIC0fmBzqofvlUwDqAZ884uaUDsJS88B0iMGAbcU5dgT0Qs4M/A='} 2023-10-03T17:56:42.621 INFO:teuthology.orchestra.console.smithi111:Power off 2023-10-03T17:56:42.621 DEBUG:teuthology.orchestra.console.smithi111:pexpect command: ipmitool -H smithi111.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-10-03T17:56:42.644 DEBUG:teuthology.orchestra.console.smithi111:power off output: Chassis Power Control: Down/Off 2023-10-03T17:56:42.645 DEBUG:teuthology.orchestra.console.smithi111:pexpect command: ipmitool -H smithi111.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-03T17:56:42.666 DEBUG:teuthology.orchestra.console.smithi111:check power output: Chassis Power is on 2023-10-03T17:56:46.667 DEBUG:teuthology.orchestra.console.smithi111:pexpect command: ipmitool -H smithi111.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-03T17:56:46.777 DEBUG:teuthology.orchestra.console.smithi111:check power output: Chassis Power is on 2023-10-03T17:56:50.781 DEBUG:teuthology.orchestra.console.smithi111:pexpect command: ipmitool -H smithi111.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-03T17:56:50.892 DEBUG:teuthology.orchestra.console.smithi111:check power output: Chassis Power is off 2023-10-03T17:56:50.992 INFO:teuthology.orchestra.console.smithi111:Power off completed 2023-10-03T17:56:51.097 INFO:teuthology.run:Summary data: description: rbd:cli/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} features/layering msgr-failures/few objectstore/bluestore-comp-zlib pool/replicated-data-pool supported-random-distro$/{ubuntu_latest} workloads/rbd_support_module_recovery} duration: 2041.2743864059448 failure_reason: 'Command failed (workunit test rbd/rbd_support_module_recovery.sh) on smithi111 with status 22: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a7de91bfa5778a0d559a7e9bc8e35f93456b41ad TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/rbd_support_module_recovery.sh''' flavor: default owner: scheduled_rraja@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=0d3bf0ae55a041efb7d9f3ffc22fb248 status: fail success: false 2023-10-03T17:56:51.098 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-03T17:56:51.161 INFO:teuthology.run:FAIL